Project Archives
This material was mostly collected from 2020 to early 2023
Background Material
- Future Challenges in Safety and Security of
Intelligent Vehicle, Jay Lala, SSIV Workshop, 24 June 2019
- Autonomous Vehicle Safety:
What Lessons Can we Learn from Aviation?, Jay Lala, 2019 International Conference on Dependable Systems & Networks
Portland, OR, USA
26 June 2019
- IVDS Project Proposal: Dependability and Security Imperatives for Intelligent Autonomous Systems, John Meyer, Summer 2019 Meeting, IFIP WG10.4, Hood River, OR
- White Paper: Intelligent Vehicle Dependability and Security, Members of the IFIP WG10.4 IVDS Project
Project Publications
- Viewpoint: Autonomous Vehicle Safety: Lessons from Aviation, Jaynarayan Lala, Carl Landwehr, John Meyer, Communications of the ACM, Sept 2020, Vol. 63, No. 9, pp. 28-31.
- ‘Move fast and break things’ won’t work for autonomous vehicles, Jaynarayan Lala, John Meyer, Carl Landwehr and Charles B. Weinstock, Opinion Contributors, The Hill, 07/21/21 2:30 PM ET
- Autonomous Vehicles: Safety Measures and Benchmarks for Perception & Cognition Functions, Jaynarayan Lala, Association for the Advancement of Artificial Intelligence (AAAI) Fall Symposium Series on Nov. 5, 2021
Events
Standards
- 24 standards and best practice guidelines were surveyed with respect to dependability and security requirements for intelligent (autonomous) road vehicles
- International: ISO (TC 22, TC 204), SAE Int’l, IEC
- USA: DOT-NHTSA, UL
- Europe: EN, BSI
- dependability (mainly safety), the most comprehensive standards are
- Safety requirements are largely qualitative with respect to both operational malfunctions of in-vehicle E/E systems (e.g., ISO 26262-3 (2018): Road vehicles - Functional Safety)
- Safety case approach of UL 4600 (2020): Standard for safety for the evaluation of autonomous products conforms with the above; focuses on full autonomy (SAE L4-5)
- ISO/PAS 21448 (2019): Road vehicles – Safety Of The Intended Functionality (SOTIF)
- security, the only road-vehicle-specific standard is still in a draft phase
- ISO/SAE DIS 21434 (2020): Road vehicles – Cybersecurity engineering
- A work in progress (e.g. ISO PAS 21448)
- Impacts safety as well (e.g., BSI PAS 11281)
- Some needs
- Quantitative measures of IV safety which, with a mix of evaluation methods and with high confidence, can assure vehicle-level quantitative requirements, e.g., for an L5 IV
- Measure: FR = number of fatal crashes caused per hour of operation Requirement: FR ≤ 5x10-8
- Better understanding of dependability-security tradeoffs in an IV context would help reconcile existing D & S standards (and lead to combined D-S standards?)
- More specific requirements (in the spirt of SOTIF) regarding safe use of ML for IV control (project goal 3)
Related Guidelines: Some Examples
- Recommended Practice reports in the SAE Int’l J-series, e.g.,
- J2980 – Considerations for ISO 26262 hazard analysis
- J3016 – Taxonomy and definitions re driving automation systems
- J3061 - Cybersecurity guidebook for cyber-physical vehicle systems
- J3101 – Hardware-protected security for ground vehicle applications
- US Dept of Transportation reports
- Recommendations by commercial consortia
Detailed List of References
1. Articles in journals, conference proceedings, and technical magazines.
- Goal and Threat Modelling for Driving Automotive Cybersecurity Risk Analysis Conforming to ISO/SAE 21434 Christophe Ponsard, Valery Ramon and Jean-Christophe Deprez. In Proceedings of the 18th International Conference on Security and Cryptography (SECRYPT 2021), pages 833-838 ISBN: 978-989-758-524-1.
- Security and Privacy Issues in Intelligent Transportation Systems: Classification and Challenges IEEE Intelligent Transportation Systems Magazine, Vol. 13, No. 1, Spring 2021.
- Cyber-Physical Security of Powertrain Systems in Modern Electric Vehicles: Vulnerabilities, Challenges and Future Visions Jin Ye, Lulu Guo, Bowen Yang, Fangyu Li, Liang Du, Le Guan, and Wenzhan Song. IEEE Journal of Emerging and Selected Topics in Power Electronics, 17 December, 2020.
- Building a self-driving car that people can trust Baidu (sponsored article), MIT Technology Review, Dec., 2020.
- Viewpoint: Self-driving vehicle technology: progress and promises. Michael A. Cusumano, Communications of the ACM, October 2020, Vol. 63, No. 10, pp. 20-22.
- Engineering of Runtime Safety Monitors for Cyber-Physical Systems with Digital Dependability Identities Jan Reich, Daniel Schneider, et al., SAFECOMP 2020, Lisbon, Portugal, Sept 2020, pp. 3-17.
- Computer Safety, Reliability, and Security Proceedings of the 39th International Conference, SAFECOMP 2020, Lisbon, Portugal, September 16–18, 2020.
- Viewpoint: Autonomous Vehicle Safety: Lessons from Aviation, Jaynarayan Lala, Carl Landwehr, John Meyer, Communications of the ACM, Sept 2020, Vol. 63, No. 9, pp. 28-31.
- Viewpoint: Unsafe At Any Level. Marc Canellas, Rachel Haga, Communications of the ACM, March 2020, Vol. 63 No. 3, Pages 31-34.
- Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy Ben Shneiderman, Int'l Journal of Human Computer Interaction, March 23, 2020.
- Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving Yulong Cao, Chaowei Xiao, Benjamin Cyr, Yimeng Zhou, Won Park, Sara Rampazzi, Qi Alfred Chen, Kevin Fu, Z. Morley Mao, CCS '19: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security November 2019 Pages 2267–2281 https://doi.org/10.1145/3319535.3339815
- Edge Computing for Autonomous Driving: Opportunities and Challenges Shaosan Liu, Liangkai Liu, Jie Tang, Bo Yu, Yifan Wang, Weisong Shi, Proceedings of the IEEE, August 2019.
- Security of Connected and Automated Vehicles Chowdhury, M., Islam, M. and Khan, Z. The Bridge. National Academy of Engineering, 49(3), pp. 46–56. Sept. 2019.
- Autonomous Vehicles: Problems and Principles for Future Regulation Jeremy A. Carp, University of Pennsylvania Journal of Law & Public Affairs, Nov. 2018.
- Robust physical-world attacks on machine learning models Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018. doi: 10.1109/CVPR.2018.00175 or https://arxiv.org/pdf/1707.08945.pdf . This is the paper with the examples showing that carefully placed pieces of tape on a stop sign can make it look like a Speed Limit 45 sign to an apparently well-trained ML algorithm.
- Current US Federal Policy Framework for Self-Driving Vehicles: Opportunities and Challenges M. J. Hanna and S. C. Kimmel, IEEE COMPUTER vol. 50, no. 12, pp. 32-40, December 2017, doi: 10.1109/MC.2017.4451211.
2. Standards, including draft standards
- UNECE Proposal for a new UN Regulation on uniform provisions concerning the approval of vehicles with regards to Automated Lane Keeping System April 20 2020 This says "proposal" but it was adopted by the UNECE in June 2020 and is effective January 1 2021.
- UL 4600 Standard for Safety for the Evaluation of Autonomous Products (Proposed) Draft of December 13, 2019.
- Standardization of Intelligent Transport Systems Standardization activities of ISO TC 204, 2019.
- ISO/PAS 21448:2019 Safety of the Intended Functionality (SOTIF), ISO TC 22, 2019.
- Provisional Standard for Autonomous Vehicles. Singapore TR-68-1. The document addresses basic behavior, safety, cybersecurity principles and assessment framework, and vehicular data types for autonomous vehicles. Available for purchase, with a free pdf "preview" at this website.
- ISO 21434 Road vehicles -- Cybersercurity Engineering Slide presentation, status overview, Angela Barber, April 2018.
3. Guidelines: Best practice guides, recommended practice guides
- US NHTSA Cybersecurity Best Practices for the Safety of Modern Vehicles Draft 2020 Update, January 12, 2021. Request for comments; comments due March 15, 2021. Federal register posting here. PDF version of Federal Register here.
- Automated Vehicle Saftey Consortium Best Practice AVSC Best Practice for Describing an Operational Design Domain: Conceptual Framework, April, 2020.
- Connected automotive ecosystems – Impact of security on safety – Code of practice British Standards Institution, December, 2018.
- SAE J3016, Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles, June, 2018.
- Automated Vehicle Research for Enhanced Safety Final Report, Submitted to the United States Department of Transportation National Highway Traffic Safety Administration (NHTSA), CAMP, AVSC, March 2016.
4. Government publications
- NHTSA Orders Crash Reporting for Vehicles Equipped with Advanced Driver Assistance Systems and Automated Driving Systems NHTSA, June 29, 2021.
- Hearing: U.S. Senate Committee on Commerce, Science and Transportation Driving Innovation: The Future of Automotive Mobility, Safety, and Technology, April 27, 2021.
- Final Report of the National Security Commission on Artificial Intelligence, March 5, 2021.
- Cybersecurity Challenges in the Uptake of Artificial Intelligence in Autonomous Driving European Union for Cybersecurity (ENISA), Feb 11, 2021. (report downloadable from linked page).
- Comments of Underwriters Laboratories on Docket No. DOT-NHTSA-2020-0106 regarding safety regulations for AVs, February 1, 2021.
- Comments of United States NTSB on Docket No. DOT-NHTSA-2020-0106 regarding safety regulations for AVs, February 1, 2021.
- US NHTSA Cybersecurity Best Practices for the Safety of Modern Vehicles Draft 2020 Update, January 12, 2021. Request for comments; comments due March 15, 2021. Federal register posting here. PDF version of Federal Register here.
- Framework for Automated Driving System Safety NHTSA Docket No. NHTSA-2020-0106, November 19, 2020.
- USDOT Automated Driving Activities April 16, 2020.
- FMVSS Considerations for Vehicles With Automated Driving Systems: Volume 1 DOT/NHTSA HS 812 796, April, 2020.
- NIST Special Publication 1900-320 Workshop Report: Consensus Safety Measurement Methodologies for Automated Driving System-Equipped Vehicles. Edward Griffor, Christopher Greer, David Wollman, Sept. 23, 2019
- Automated Vehicle Safety Expert Panel: Engaging Drivers and Law Enforcement Governors Highway Safety Association (USA), August, 2019.
- The National Artificial Intelligence R&D Strategic Plan: 2019 Update US National Science and Technology Council, June 21,2019.
- The German PEGASUS Project Symposium. This is a project with 17 German industrial partners and the goals of (1) defining a standardized procedure for the testing and experimenting of automated vehicle systems in simulation, on test stands and in real environments, (2) Developing a continuous and flexible tool chain to safeguard automated driving (2) Integrating the tests in the development processes at an early stage (4) Creating a cross-manufacturer method for safeguarding highly automated driving functions. May 14, 2019.
- Release announcement for Singapore Technical Reference TR 68 Singapore's provisional standard for fully autonomous vehicles, January, 2019.
- UN Economic Commission for Europe (UNECE) website on automated driving
- Preparing for the Future of Transportation: Automated Vehicles 3.0 (AV 3.0), USDOT, September 28, 2018.
- China Guideline for developing national internet of vehicles industry standard system January, 2018.
- Automated Driving Systems 2.0: A Vision for Safety US National Highway Traffic Safety Administration (NHTSA) September, 2017.
- Automated and Connected Driving. Ethics Commission, German Federal Ministry of Transport and Digital Infrastructure. Report (Extract) June, 2017.
- Cybersecurity best practices for modern vehicles, National Highway Traffic Safety Administration (NHTSA), Report No. DOT HS 812 333, October, 2016.
5. Blogs, and articles from the technical press
- NTSB asks NHTSA for more self-driving car rules citing Tesla's full self-driving beta Kyle Hyatt, CNN Road/Show, March 12, 2021.
- First Commercial Autonomous Bus Services Hit Singapore Roads Eileen Yu, ZDNet, January 25, 2021
- A Users Guide to Vehicle Automation Modes Phil Koopman, Edge Case Research, January 28, 2021
- Highway Safety Agency Wants Car Makers to Know What’s in Their Software Mariam Baksh, NextGov, January 15, 2021.
- Countries agree regulations for automated driving Japan Times, June 26, 2020.
- ISO 26262 & Model-Based Design: A Match Made in Heaven? Embitel blog, May 9, 2020.
- Multiple standards to emerge in 2020 for AI-driven vehicles Junko Yoshida, Embedded blog, January 20, 2020.
- Safe Autonomy Phil Koopman, Blog.
- Study measures how fast humans react to road hazards Rob Mathieson, MIT News office, August 2019.
- The race toward autonomous driving Interview with Herbert Diess, until June 8m, 2020, CEO of Volkswagen.
- Dossier: The leaders in self-driving cars by Alison DeNisco, ZDNet, Feb. 1, 2018.
6. Articles from general circulation newspapers and magazines
- Gatik’s Driverless Trucks Set To Scale For Walmart, Launching In 12 Markets, Forbes, November 8, 2021.
- Companies are racing to make self-driving cars. But why?, Washington Post, February 4, 2022.
- Carmakers Strive to Stay Ahead of Hackers, New York Times, March 18, 2021.
- Tesla Autopilot and Other Driver-Assist Systems Linked to Hundreds of Crashes, New York Times, June 15, 2022.
- California regulators approve state’s 1st robotic taxi fleet, AP News, June 2, 2022.
- Self-Driving Cars Are Going Nowhere, Max Chafkin, Bloomberg Businessweek, October 6, 2022.
- Aurora releases tool to gauge safety of self-driving systems Paul Lienert, Reuters, August 18, 2021. Aurora's announcement from their website
- Active Driving Assistance Systems: Test Results and Design Recommendations. Consumer Reports, November 2020.
- Honda says will be first to mass produce level 3 autonomous cars Reuters, Nov. 11, 2020.
- FAA gives preliminary approval on design fixes for 737 Max by Ian Duncan and Lori Aratani, Washington Post, August 3, 2020.
- Can Automated Vehicles Prove Themselves to Be Safe? by Marjory Blumenthal and Laura Fraade-Blanar, Issues in Science and Technology (US National Academies), VOL. XXXVI, No. 4, Summer 2020. (See RAND tech report from Nov. 2018 cited in Tech Reports section below)
- Countries agree regulations for automated driving the Japan Times, June 26, 2020.
- Study: Autonomous vehicles won’t make roads completely safe Tom Krisher, Associated Press, June 4, 2020.
- Video shows Tesla on autopilot slam into truck on Taiwan highway by Keoni Everington, Taiwan News, June 2, 2020.
- A case for cooperation between humans and machines by John Markoff, New York Times, May 21, 2020
- Ex-Uber self-driving head Levandowski declares bankruptcy after $179 million loss to Google CNBC/Reuters, March 4, 2020.
- Amazon-Backed Aurora Readies An Air-Traffic Control System For Self-Driving Cars by Alan Ohnsman, Forbes, December 16, 2019.
- Five components of autonomous car security by Alexander Polyakov, Forbes, October 31. 2019.
- Discussing "Safety First for Automated Driving" with Aptiv's Karl Iagnemma by Lance Eliot, Forbes, July, 19, 2019.
- Key To Driverless Cars, Operational Design Domains (ODD), Here’s What They Are, Woes Too, by Lance Eliot, Medium.com, April 19,2019.
- Ford rethinks Level 3 autonomy by Michael Martinez, Automotive News Europe, January 20, 2019.
- Few carmakers submit self-driving safety assessments Detroit News, September 10, 2018.
- Interview with Chris Urmson, CEO of Aurora The Atlantic, March 29, 2018.
7. Videos, podcasts, lectures, demonstrations, etc.
- Are Autonomous Vehicles A Grand Challenge for CPS or AI?] Prof. Raj Rajkumar, slides of the keynote at NSF CPS PI Meeting, June 4, 2021.
- A New Approach to Engineering for Safety and Security Prof. Nancy Leveson, slides of Zoom seminar to Univ. of Michigan Dept. of Aerospace Engineering, Oct. 8, 2020.
- Future pitfalls and promises of safety in autonomous systems Prof. Missy Cummings, Zoom lecture to NTSB. 75 minutes. May 22, 2020.
- AI for Full-Self Driving Andrej Karpathy, lecture, 30 minutes, ScaledML Conference, Feb 26-27, 2020.
- Tesla Autopilot and Multi-Task Learning for Perception and Prediction Lecture, 24 minutes, clip from ICML, June 2019, workshop on multitasking.
- ISO-21434 Autonomous Vehicle Safety and Security (podcast series) Hassan Akram. Discussions of ISO 21434.
8. Books, technical reports, websites, and other references
- The Airline Safety
Revolution, The Wall Street Journal, April 16, 2021
- How self-driving cars got stuck in the slow lane, The Guardian, March 27, 2022
- Federal safety regulators begin
investigation into self-driving
taxis, SMARTCITIESDIVE, December 22, 2022
- Tesla recalling nearly 363,000 vehicles equipped with ‘Full Self-Driving’, cnn.com, February 16, 2023
- Issues in Autonomous Vehicle Testing and Deployment Bill Canis, Congressional Research Service Report R45985, Apr. 23, 2021.
- Web site of TheAutonomous series of conferences.
- Safe Enough: Approaches to Assessing Acceptable Safety for Automated Vehicles Marjory S. Blumenthal, Laura Fraade-Blanar, Ryan Best, J. Luke Irwin, Rand Report RR-A569-1, Oct. 2020.
- DARPA project aimed at protecting vehicles from hackers by Gabe Cherry, Michigan Engineering, August 24, 2020.
- Chapter 27 Assurance and Sustainability Ross Anderson, Security Engineering, 3rd. Ed. (draft), pp. 859-896, July 22, 2020. Note especially Section 27.4 p.884 ff., "The Entanglement of Safety and Security" which addresses automotive electronics.
- BMW Releases Safety Assessment Report of Autonomous Vehicle Testing May 20, 2020
- Cyberphysical Constructs and Concepts for Fully Automated Networked Vehicles Gerard Le Lann, Technical Report, October 2019.
- Safety First For Automated Driving , Technical Report, July 2019.
- Measuring Automated Vehicle Safety: Forging a Framework Laura Fraade-Blanar, Marjory S. Blumenthal, James M. Anderson, Nidhi Kalra, RAND Corporation, Nov. 2018.
- On a Formal Model for Safe and Scalable Self-Driving Cars Shai Shalev-Schwartz, Shaked Shammah, Amnon Shashua, Intel Mobileye, v6 27 October, 2018.
- Technology and Consciousness Daniel Sanchez and John Rushby, Workshops Report, SRI International, 30 September, 2018.
- Waymo Safety Report: On the Road to Fully Self-Driving Google/Waymo 2018.
- General Motors Self-Driving Safety Report General Motors,2018.
- Appendix A to the above report General Motors, 2018.
- Chapter in: Functional Safety of Automated Driving Systems Does ISO 26262 Meet the Challenges?, Helmut Martin, Kurt Tschabuschnig, Olof Bridal, Daniel Watzenig, 2017.
- Driving to Safety, How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability? Nidhi Kalra, Susan M. Paddock, RAND Report 1478, 2016.
|
|