Human agency has become increasingly limited in complex systems with increasingly automated decision-making capabilities. For instance, human occupants are passengers and do not have direct vehicle control in fully automated cars (i.e., driverless cars). An interesting question is whether users are responsible for the accidents of these cars. Normative ethical and legal analyses frequently argue that individuals should not bear responsibility for harm beyond their control. Here, we consider human judgment of responsibility for accidents involving fully automated vehicles through three studies with seven experiments (N = 2668). We compared the responsibility attributed to the occupants in three conditions: an owner in his private fully automated car, a passenger in a driverless robotaxi, and a passenger in a conventional taxi, where none of these three occupants have direct vehicle control over the involved vehicles that cause identical pedestrian injury. In contrast to normative analyses, we show that the occupants of driverless cars (private cars and robotaxis) are attributed more responsibility than conventional taxi passengers. This dilemma is robust across different contexts (e.g., participants from China vs the Republic of Korea, participants with first- vs third-person perspectives, and occupant presence vs absence). Furthermore, we observe that this is not due to the perception that these occupants have greater control over driving but because they are more expected to foresee the potential consequences of using driverless cars involving counterfactual thinking. Our findings suggest that when driverless vehicles (private cars and taxis) cause harm, their users may face more social pressure, which public discourse and legal regulations should manage appropriately.