From 51b3553ee4745dbb6a049715458ef41bab29277f Mon Sep 17 00:00:00 2001 From: Linyan Huang Date: Tue, 5 Mar 2024 09:30:46 +0000 Subject: [PATCH 1/6] Update sampled data --- challenge/llama-adapter-DriveLM.json | 324 +++++- challenge/test.json | 756 ++++++++++++++ challenge/test_eval.json | 982 ++++++++++++++++++ challenge/test_llama.json | 1388 ++++++++++++++++++++++++++ challenge/test_v1.json | 226 ----- challenge/test_v2.json | 317 ------ 6 files changed, 3441 insertions(+), 552 deletions(-) create mode 100644 challenge/test_eval.json create mode 100644 challenge/test_llama.json delete mode 100644 challenge/test_v1.json delete mode 100644 challenge/test_v2.json diff --git a/challenge/llama-adapter-DriveLM.json b/challenge/llama-adapter-DriveLM.json index 625c561..7ba5c7b 100644 --- a/challenge/llama-adapter-DriveLM.json +++ b/challenge/llama-adapter-DriveLM.json @@ -7,9 +7,9 @@ }, { "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_1", - "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Turn right. C. Turn left. D. Stopped.", - "gt_answer": "C", - "answer": "The moving status of object is \"Stopped.\"" + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Stopped. C. Back up. D. Turn left.", + "gt_answer": "D", + "answer": "The moving status of object is \"Back up.\"" }, { "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_2", @@ -39,13 +39,13 @@ "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_6", "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", "gt_answer": "Keep going at the same speed, decelerate gradually without braking.", - "answer": "In the image, the ego vehicle is driving through a rainy city street with a traffic light. To ensure safety, the driver should follow these actions:1. Maintain a safe distance from other vehicles: Keep a safe distance from other vehicles, especially if visibility is low due to the rain.2. Reduce speed: Slow down to allow for better visibility and to avoid hydroplaning.3. Use headlights: Turn on headlights to increase visibility for both the driver and other road users.4. Be cautious at intersections: Be extra vigilant at intersections, as visibility may be reduced, and other drivers may not be able to see you clearly.5. Be prepared for sudden stops: Anticipate sudden stops or changes in traffic conditions, and be prepared to react quickly.6. Avoid distractions: Avoid using cell phones, eating, or other distractions while driving in the rain, as these can impair your ability to react to changing road conditions.7. Be patient: Allow other vehicles to merge or change lanes, and be prepared to yield to pedestrians or cyclists who may be more vulnerable in the rain.By following these safety measures" + "answer": "In the image, the ego vehicle is driving through a rainy city street with a traffic light. To ensure safety, the driver should follow these actions:1. Maintain a safe distance from other vehicles: Keep a safe distance from other vehicles, especially if visibility is low due to the rain.2. Reduce speed: Slow down to allow for better visibility and to avoid hydroplaning.3. Use headlights: Turn on headlights to increase visibility for both the driver and other road users.4. Be cautious at intersections: Be extra vigilant at intersections, as visibility may be reduced, and other drivers may not be able to see you clearly.5. Be prepared for sudden stops: Anticipate the need to stop suddenly due to wet roads, and be prepared to react quickly.6. Avoid distractions: Avoid using cell phones, eating, or other distractions while driving in the rain, as these activities can divert attention from the road.7. Be patient: Allow other vehicles to pass, and be prepared to wait for traffic lights to change.By following these safety measures, the ego vehicle driver can ensure a safer and more comfortable driving experience in" }, { "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_7", - "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is steering to the right. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving fast.", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed.", "gt_answer": "B", - "answer": "The ego vehicle is slightly steering to the right, and it is driving slowly." + "answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed." }, { "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_0", @@ -57,7 +57,7 @@ "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_1", "question": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", "gt_answer": "Firstly, notice . The object is a traffic sign, so the ego vehicle should continue at the same speed. Secondly, notice . The object is a traffic sign, so the ego vehicle should accelerate and continue ahead. Thirdly, notice . The object is stationary, so the ego vehicle should continue ahead at the same speed.", - "answer": "1. The ego vehicle should notice the traffic light first, as it is the first object in the image. The traffic light is in the process of changing from red to green, indicating that the vehicle should proceed with caution.2. The ego vehicle should notice the truck next, as it is the second object in the image. The truck is stopped at the intersection, and the ego vehicle should be cautious and wait for the truck to move before proceeding.3. The ego vehicle should notice the bus next, as it is the third object in the image. The bus is stopped at the intersection, and the ego vehicle should be cautious and wait for the bus to move before proceeding.4. The ego vehicle should notice the traffic light next, as it is the fourth object in the image. The traffic light is now green, indicating that the vehicle can proceed safely." + "answer": "1. The ego vehicle should notice the traffic light first, as it is the first object in the image. The traffic light is in the process of changing from red to green, indicating that the vehicle should proceed with caution.2. The ego vehicle should notice the truck next, as it is the second object in the image. The truck is stopped at the intersection, and the ego vehicle should be cautious when approaching the intersection to ensure it does not collide with the truck.3. The ego vehicle should notice the bus next, as it is the third object in the image. The bus is stopped at the intersection, and the ego vehicle should be cautious when approaching the intersection to ensure it does not collide with the bus.4. The ego vehicle should notice the traffic light next, as it is the fourth object in the image. The traffic light is in the process of changing from green to red, indicating that the vehicle should stop at the intersection.5. The ego vehicle should notice the truck next, as it is the fifth object in the image. The truck is stopped at the intersection, and the ego vehicle should be cautious when approaching the intersection to ensure it" }, { "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_2", @@ -85,8 +85,314 @@ }, { "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_6", - "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is steering to the left. The ego vehicle is driving fast. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is going straight. The ego vehicle is driving fast.", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. B. The ego vehicle is steering to the left. The ego vehicle is driving fast. C. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving slowly.", + "gt_answer": "D", + "answer": "The ego vehicle is driving with normal speed and is steering to the left." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a white sedan to the front left of the ego vehicle, a gray sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and .", + "answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a truck, a bus, a car, a traffic light, and a street sign. The presence of these objects indicates that the scene is likely a busy intersection or a city street, where vehicles are moving and interacting with each other. The traffic light and street sign provide essential information for drivers to navigate the area safely and efficiently. The truck and bus are also important, as they are large vehicles that require extra attention and care when sharing the road. The car is a smaller vehicle that may also be affected by the presence of the larger vehicles. Overall, these objects will be considered for the future reasoning and driving decision, as they play a crucial role in ensuring safe and efficient traffic flow in the area." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Going ahead. C. Turn right. D. Turn left.", + "gt_answer": "B", + "answer": "The moving status of object is \"Stopped.\"" + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_2", + "question": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "gt_answer": "Firstly notice that ; the object is a traffic sign, so the ego vehicle should continue going ahead at the same speed. Secondly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed. Thirdly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed.", + "answer": "1. The ego vehicle should notice the traffic light first, as it is the first object in the image. The traffic light is in the process of changing from red to green, indicating that the vehicle should proceed with caution.2. The ego vehicle should notice the truck next, as it is the second object in the image. The truck is stopped at the intersection, and the ego vehicle should be cautious and wait for the truck to move before proceeding.3. The ego vehicle should notice the traffic light again, as it is the third object in the image. The traffic light is now green, indicating that the vehicle can proceed.The ego vehicle should take the appropriate action based on the state of the traffic light and the position of the other vehicles on the road. If the traffic light is red, the ego vehicle should stop and wait for the light to change. If the traffic light is green, the ego vehicle should proceed with caution, being mindful of other vehicles and pedestrians in the area." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_3", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The traffic sign is located on the side of the road, while the road barrier is located in the middle of the road." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_4", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to keep going at the same speed, the reason is that there is no safety issue. The probability of this action is high.", + "answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way. Turning left would allow the vehicle to continue its journey on the road, avoiding any potential conflicts with other vehicles or pedestrians." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_5", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "No such action will lead to a collision.", + "answer": "The ego vehicle can lead to a collision with by not following the traffic rules or driving in an unsafe manner. Some actions that can lead to a collision include:1. Failing to stop at a red light or stop sign, as seen in the image.2. Driving too fast for the current road conditions, such as driving through a puddle or rain.3. Failing to maintain a safe distance from other vehicles, especially when approaching a stop sign or intersection.4. Failing to yield the right of way to pedestrians or other vehicles at a crosswalk or intersection.5. Driving distracted or under the influence of alcohol or drugs.By taking these actions, the ego vehicle can put itself and other road users at risk of a collision. It is essential for drivers to be aware of their surroundings, follow traffic rules, and drive responsibly to ensure the safety of themselves and others on the road." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_6", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Keep going at the same speed and brake suddenly.", + "answer": "In the image, the ego vehicle is driving down a wet street, which can be a challenging situation due to the wet road conditions. To ensure safety, the driver should take the following actions:1. Slow down: Driving at a slower speed helps maintain better control of the vehicle and reduces the risk of skidding or hydroplaning.2. Keep a safe distance: Maintain a safe distance from other vehicles and obstacles, as the wet road conditions can make it more difficult to stop in time.3. Use appropriate tires: Ensure that the vehicle is equipped with appropriate tires for wet conditions, such as all-season or winter tires, which have better traction and grip on wet surfaces.4. Be cautious of pedestrians: Be extra vigilant for pedestrians, as they may be more difficult to see in the rain.5. Avoid sudden braking or acceleration: Sudden changes in speed or direction can cause the vehicle to lose traction, leading to a loss of control.By following these safety measures, the ego vehicle driver can minimize the risk of accidents and ensure a safe journey on the wet street." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_7", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. B. The ego vehicle is going straight. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is not moving. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.", + "gt_answer": "B", + "answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a golden sedan to the back of the ego vehicle, a black sedan car to the back of the ego vehicle, and a white truck to the back of the ego vehicle. The IDs of these objects are , , and .", + "answer": "In the current scene, there are several important objects that need to be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, a street sign, and a traffic light. The presence of these objects indicates that the scene is a busy urban area with multiple vehicles and pedestrians. The traffic light is an essential element for controlling the flow of traffic and ensuring safety. The street sign provides information about the location and direction of the street, which is crucial for navigation and understanding the context of the scene. The presence of a truck and a bus indicates that the area is likely a commercial or transportation hub, with vehicles carrying goods or passengers. The car is a common sight in urban areas, and its presence indicates that the scene is a typical urban environment with various modes of transportation. The combination of these objects creates a complex and dynamic environment that drivers need to navigate and adapt to." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Turn left. C. Going ahead. D. Turn right.", + "gt_answer": "C", + "answer": "The moving status of object is \"Going ahead.\"" + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_2", + "question": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "gt_answer": "Firstly notice that , the object is turning left, so the ego vehicle should keep going ahead at the same speed. Secondly notice that , the object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that , it is stationary, so the ego vehicle should keep going ahead at the same speed.", + "answer": "1. The ego vehicle should notice the traffic light first when getting to the next possible location. The traffic light is in the foreground of the image, and it is the first object that the ego vehicle should notice.2. The state of the traffic light is red, indicating that the ego vehicle should stop at the intersection. The ego vehicle should come to a complete stop at the red light, as per traffic rules and regulations.3. The ego vehicle should notice the street sign next. The street sign is located behind the traffic light, and it is the second object that the ego vehicle should notice.4. The state of the street sign is unclear, as it is partially obscured by the traffic light. The ego vehicle should pay attention to the street sign and ensure that it is visible before proceeding.5. The ego vehicle should notice the car in the background. The car is located behind the street sign, and it is the third object that the ego vehicle should notice.6. The state of the car in the background is unclear, as it is partially obscured by the street sign. The ego vehicle should pay attention to the car in the background and ensure that it is visible before proceeding." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_3", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The image shows a traffic sign, not a road barrier." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_4", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to keep going at the same speed because there is no safety issue. The probability of taking this action is high.", + "answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way. Turning left would allow the vehicle to continue its journey on the road, avoiding any potential collisions with other vehicles or pedestrians." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_5", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "No such action will lead to a collision.", + "answer": "The ego vehicle can lead to a collision with by not paying attention to the traffic situation and failing to maintain a safe distance from the other vehicle. The ego vehicle should be aware of the other vehicle's position and speed, and adjust its speed and course accordingly to avoid a collision. This includes maintaining a safe following distance, slowing down or stopping if necessary, and being prepared to react to sudden changes in the other vehicle's behavior." + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_6", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Keep going at the same speed, accelerate, and proceed ahead.", + "answer": "In the image, the ego vehicle is driving through a rainy city street. To ensure safety, the driver should take the following actions:1. Maintain a safe distance from other vehicles: The wet road conditions can make it more difficult to stop or maneuver, so it is essential to maintain a safe distance from other vehicles.2. Reduce speed: Driving at a slower speed helps to maintain better control of the vehicle and allows for more time to react to any sudden changes in the road or traffic conditions.3. Use headlights: Turn on the headlights to increase visibility for both the driver and other road users.4. Be cautious of pedestrians: The wet road conditions may make it more difficult for pedestrians to maintain their footing, so the driver should be extra vigilant and slow down when approaching crosswalks or intersections.5. Avoid sudden braking or acceleration: Sudden changes in speed or direction can cause the vehicle to lose traction, leading to skidding or hydroplaning.6. Be prepared for puddles: The driver should be prepared for puddles and other wet spots on the road, as they may cause" + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_7", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. C. The ego vehicle is going straight. The ego vehicle is driving fast. D. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed.", "gt_answer": "C", - "answer": "The ego vehicle is driving fast, as it is steering to the left and going straight." + "answer": "The ego vehicle is slightly steering to the left, and it is driving slowly." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a red car to the front of the ego vehicle, a white SUV to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a traffic light, a street sign, a building, and a car. The traffic light is located at the intersection, and it is important for drivers to pay attention to its position and color to ensure they follow the traffic rules and regulations. The street sign provides information about the street name, which is essential for drivers to navigate the area. The building is situated near the intersection, and it could be a landmark or a business establishment, which drivers should be aware of when navigating the area. The car is parked on the side of the road, and it could be a potential obstacle or hazard for other drivers. The presence of these objects in the scene will be considered by the AI system while making driving decisions, ensuring the safety and efficiency of the road network." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Back up. C. Turn left. D. Going ahead.", + "gt_answer": "D", + "answer": "The moving status of object is \"Stopped.\"" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_2", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The image description does not provide enough information to definitively determine whether the object is a traffic sign or a road barrier. However, it is possible that it could be either one, depending on the context and the specific details of the object." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_3", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to brake gently to a stop. The reason is to follow the traffic rules, and the probability is high.", + "answer": "Based on the given coordinates, the ego vehicle could take the action of turning left at the intersection. The probability of this action is high, as the vehicle is already at the intersection and has the right-of-way to turn left. The vehicle is likely to turn left to continue its journey, as it is already in the intersection and has the right-of-way." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_4", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "Accelerate and go straight.", + "answer": "The ego vehicle can lead to a collision with by:1. Failing to stop at the red traffic light: If the ego vehicle fails to stop at the red traffic light, it may collide with the other vehicle that is already stopped at the intersection.2. Failing to yield to the other vehicle: If the ego vehicle fails to yield to the other vehicle that is already stopped at the intersection, it may collide with the vehicle that is waiting to turn left.3. Turning left without checking for oncoming traffic: If the ego vehicle turns left without checking for oncoming traffic, it may collide with the other vehicle that is already stopped at the intersection.4. Turning left without checking for pedestrians: If the ego vehicle turns left without checking for pedestrians, it may collide with pedestrians crossing the street.In summary, the ego vehicle can lead to a collision with by failing to stop at the red traffic light, failing to yield to the" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_5", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Brake gently to a stop, brake suddenly, and slightly offset to the left.", + "answer": "In the image, the ego vehicle is driving down a street with a traffic light at the intersection. To ensure safety, the ego vehicle should follow the traffic light's instructions and obey the traffic rules. This includes stopping at red lights, yielding to pedestrians and other vehicles, and maintaining a safe distance from other vehicles. Additionally, the driver should be aware of their surroundings, including pedestrians, cyclists, and other vehicles, and be prepared to react to any unexpected situations. By following these safety measures, the ego vehicle can help prevent accidents and maintain a smooth flow of traffic in the area." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_6", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed.", + "gt_answer": "C", + "answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a white sedan car to the front of the ego vehicle, a blue sedan to the front right of the ego vehicle, a white sedan to the front right of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a traffic light, a car, a building, and a street sign. The traffic light is located at the intersection, and it is important for drivers to pay attention to its position and color to determine whether they can proceed or need to stop. The car is driving on the road, and it is essential for the driver to be aware of its position and speed to avoid collisions or accidents. The building is situated near the intersection, and it may be a landmark or a point of interest for the driver. The street sign provides information about the street name, which can be helpful for navigation or identifying the location. Overall, these objects play a crucial role in the driver's decision-making process and contribute to the overall safety and efficiency of the road system." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Turn right. D. Stopped.", + "gt_answer": "C", + "answer": "The moving status of object is \"Stopped.\"" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_2", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The traffic sign is located on the side of the road, and it is not a road barrier." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_3", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to remain stationary. The reason for this action is to follow the traffic rules. The probability of this action is high.", + "answer": "Based on the given coordinates, the ego vehicle is located at the intersection of two streets, with a traffic light in front of it. The ego vehicle could take several actions, such as:1. Continue straight: The ego vehicle could continue driving straight through the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and the traffic light is indicating that it should proceed.2. Turn left: The ego vehicle could turn left at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and the traffic light is indicating that it should proceed.3. Turn right: The ego vehicle could turn right at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and the traffic light is indicating that it should proceed.4. Stop: The ego vehicle could stop at the intersection, waiting for the traffic light to change or for other vehicles to pass before proceeding. This action has a probability of 0% as the vehicle is already at the intersection and the traffic" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_4", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "Accelerating and going straight.", + "answer": "The ego vehicle can lead to a collision with by failing to yield to the traffic light, which is red, and proceeding through the intersection. This can result in a collision with the other vehicle, which is waiting at the intersection." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_5", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Brake gently to a stop, slightly offset to the right, and slightly offset to the left.", + "answer": "In the image, the ego vehicle is approaching a traffic light at an intersection. To ensure safety and follow traffic rules, the ego vehicle should:1. Stop at the red light, if it is currently red, and wait for the green light to proceed.2. If the light is green, the ego vehicle should proceed cautiously, checking for any pedestrians or other vehicles that might be crossing the intersection.3. If the light turns yellow, the ego vehicle should slow down and prepare to stop if necessary.4. If the light turns red, the ego vehicle should stop completely and wait for the light to turn green again before proceeding.5. If the ego vehicle is turning left or right at the intersection, it should yield to any oncoming traffic and wait for a safe gap to make the turn.6. If the ego vehicle is turning left or right at the intersection, it should yield to any oncoming traffic and wait for a safe gap to make the turn.7. If the ego vehicle is turning left or right at the intersection, it should yield to any oncoming traffic and wait for a safe gap to make the turn.8. If the ego vehicle is turning left or right" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_6", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is steering to the right. The ego vehicle is driving slowly. D. The ego vehicle is steering to the right. The ego vehicle is driving fast.", + "gt_answer": "A", + "answer": "The ego vehicle is steering to the right, and it is driving fast." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a white truck to the front right of the ego vehicle, a white sedan to the front of the ego vehicle, a gray sedan to the front of the ego vehicle, a white sedan to the front right of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, a traffic light, and a building. The car and truck are both on the road, and the bus is also visible on the street. The traffic light is located at the intersection, and the building is situated near the intersection as well. These objects will be important for the driver to consider when making decisions about their driving path, speed, and potential hazards." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Going ahead. D. Turn left.", + "gt_answer": "C", + "answer": "The moving status of object is \"Going ahead.\"" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_2", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The traffic sign is located on the side of the road, while the road barrier is located in the middle of the road." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_3", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to remain stationary. The reason for taking this action is to follow traffic rules. The probability of this action is high.", + "answer": "Based on the given coordinates, the ego vehicle is located at the intersection of two streets, with a traffic light on the left side of the road. The vehicle is facing the traffic light, which is green. The ego vehicle could take several actions based on this situation. It could:1. Continue straight through the intersection, following the green traffic light, as it is currently allowed to do so.2. Turn left at the intersection, following the green traffic light, as it is currently allowed to do so.3. Turn right at the intersection, following the green traffic light, as it is currently allowed to do so.4. Stop at the intersection, as it is currently allowed to do so.The probability of the ego vehicle taking any of these actions depends on the specific traffic rules and regulations in the area. In general, the probability of the ego vehicle turning left or right at the intersection is higher than continuing straight through, as it is more likely that the vehicle is turning at an intersection rather than continuing straight. However, the exact probability of each action depends on the specific traffic rules and regulations in the area." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_4", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "Turning right and accelerating.", + "answer": "The ego vehicle can lead to a collision with by not following the traffic rules, such as speeding, not yielding to pedestrians or other vehicles, or failing to maintain a safe distance from the vehicle in front. Additionally, if the ego vehicle is not paying attention to the road or is distracted, it can also contribute to a collision." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_5", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Brake gently to a stop, slightly offset to the left.", + "answer": "In the image, the ego vehicle is driving down a street with a traffic light at the intersection. To ensure safety, the ego vehicle should follow the traffic light's instructions and obey the traffic rules. This includes stopping at red lights, yielding to pedestrians and other vehicles, and maintaining a safe distance from other vehicles. Additionally, the driver should be aware of their surroundings, including pedestrians, cyclists, and other vehicles, and be prepared to react to any unexpected situations. By following these safety measures, the ego vehicle can help prevent accidents and maintain a smooth flow of traffic in the city." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_6", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is steering to the right. The ego vehicle is driving very fast.", + "gt_answer": "A", + "answer": "The ego vehicle is steering to the left, and it is driving slowly." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a white truck to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , and .", + "answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, and a traffic light. The car and truck are both on the road, and the bus is parked on the side of the road. The traffic light is located at the intersection, indicating that the vehicles need to follow the traffic rules and regulations. The presence of these objects and their positions in the scene will be taken into account by the driver when making future decisions, such as when to change lanes, when to stop, or when to proceed through the intersection." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Turn left. C. Stopped. D. Back up.", + "gt_answer": "A", + "answer": "The moving status of object is \"Stopped.\"" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_2", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The traffic sign is located on the side of the road, and it is not a road barrier." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_3", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to remain stationary, the reason is to follow the traffic rules, high probability.", + "answer": "Based on the given coordinates, the ego vehicle could take the following actions:1. Continue driving straight ahead: The vehicle is currently driving on a road, and the coordinates indicate that it is at the intersection of two roads. The vehicle could continue driving straight ahead, following the road it is currently on.2. Turn left: The vehicle could also turn left at the intersection, following the road that is perpendicular to the one it is currently on. This would require the vehicle to make a left turn at the intersection.3. Turn right: The vehicle could also turn right at the intersection, following the road that is parallel to the one it is currently on. This would require the vehicle to make a right turn at the intersection.The probability of the vehicle taking any of these actions depends on the specific situation and the driver's preferences. For example, if the vehicle is approaching a busy intersection or a blind spot, it might be more likely to turn left or right to avoid potential collisions or to maintain a safe distance from other vehicles." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_4", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "Moderate right turn.", + "answer": "The ego vehicle can lead to a collision with by not following the traffic rules, such as speeding, not stopping at a red light, or not yielding to pedestrians or other vehicles. Additionally, if the ego vehicle is not paying attention to its surroundings or is distracted, it could also contribute to a collision." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_5", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Brake gently to a stop, slightly offset to the right.", + "answer": "In the image, the ego vehicle is driving down a street with a green light. It is important for the ego vehicle to follow the traffic rules and regulations, such as stopping at the red light when it turns red. Additionally, the ego vehicle should be cautious of other vehicles, pedestrians, and cyclists in the area, as there are multiple people and vehicles present. The ego vehicle should maintain a safe distance from other vehicles and be prepared to stop or slow down if necessary. It is also essential to be aware of any potential hazards or obstacles on the road, such as potholes or debris, and adjust the speed accordingly to ensure a safe journey." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_6", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is not moving. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.", + "gt_answer": "B", + "answer": "The ego vehicle is going straight and driving with normal speed." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_0", + "question": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "gt_answer": "There is a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , and .", + "answer": "In the current scene, there are several important objects that will be considered for the future reasoning and driving decision. These objects include a car, a truck, a bus, a traffic light, and a building. The car and truck are both on the road, and the bus is parked on the side of the road. The traffic light is located at the intersection, and the building is situated in the background. These objects will be important for the driver to consider when navigating the road, ensuring safety, and making decisions about the traffic flow." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_1", + "question": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Turn left. B. Turn right. C. Back up. D. Going ahead.", + "gt_answer": "D", + "answer": "The moving status of object is \"Going ahead.\"" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_2", + "question": "\nIs a traffic sign or a road barrier?", + "gt_answer": "No.", + "answer": "The image description does not provide enough information to definitively determine whether the object is a traffic sign or a road barrier. However, it is possible that it could be either one, depending on the context and the specific details of the object." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_3", + "question": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "gt_answer": "The action is to keep going at the same speed, the reason being that there is no safety issue. The probability of this action is high.", + "answer": "Based on the given coordinates, the ego vehicle is located at the intersection of two streets, with a traffic light on the left side of the road. The ego vehicle could take several actions, such as:1. Continue straight: The ego vehicle could continue driving straight through the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and following the traffic light's instructions.2. Turn left: The ego vehicle could turn left at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and following the traffic light's instructions.3. Turn right: The ego vehicle could turn right at the intersection, following the traffic light's instructions. This action has a probability of 100% as the vehicle is already at the intersection and following the traffic light's instructions.4. Stop: The ego vehicle could stop at the intersection, waiting for the traffic light to change or for other vehicles to pass. This action has a probability of 0% as the vehicle is already at the intersection and following the traffic light's instructions." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_4", + "question": "\nWhat actions taken by the ego vehicle can lead to a collision with ?", + "gt_answer": "Accelerating and going straight can lead to a collision with .", + "answer": "The ego vehicle can lead to a collision with by:1. Failing to yield to the pedestrian crossing the street: If the ego vehicle is not paying attention to the pedestrian crossing the street, it may not notice the pedestrian in time and could collide with them.2. Failing to stop at the stop sign: If the ego vehicle fails to stop at the stop sign, it may continue driving without checking for oncoming traffic or pedestrians, increasing the risk of a collision.3. Failing to yield to the traffic light: If the ego vehicle fails to yield to the traffic light, it may continue driving without checking for oncoming traffic or pedestrians, increasing the risk of a collision.4. Failing to maintain a safe distance from the vehicle in front: If the ego vehicle is too close to the vehicle in front, it may not have enough time to react to sudden changes in the vehicle's speed or direction, increasing the risk of a collision.5. Driving too fast for the road conditions: If the ego vehicle is driving too" + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_5", + "question": "\nIn this scenario, what are safe actions to take for the ego vehicle?", + "gt_answer": "Brake gently to a stop, slightly offset to the left.", + "answer": "In the image, the ego vehicle is driving down a street with a sidewalk on the side. To ensure safety, the driver should:1. Maintain a safe distance from the sidewalk, especially if pedestrians are present.2. Be cautious of any pedestrians or cyclists who may be crossing the street or using the sidewalk.3. Obey traffic signals and signs, as well as any posted speed limits.4. Be aware of any potential hazards or obstacles on the road, such as potholes, debris, or other vehicles.5. Be prepared to stop or slow down if necessary to avoid collisions or accidents.By following these safety measures, the driver can help ensure the safety of themselves, their passengers, and other road users." + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_6", + "question": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed.", + "gt_answer": "D", + "answer": "The ego vehicle is slightly steering to the left, and it is driving with normal speed." } ] \ No newline at end of file diff --git a/challenge/test.json b/challenge/test.json index e458e0c..da3342d 100644 --- a/challenge/test.json +++ b/challenge/test.json @@ -220,6 +220,762 @@ "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" } + }, + "d9075c2a5f864a2b8abf41e703f4cf1c": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white sedan to the front left of the ego vehicle, a gray sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Going ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "A": "Firstly notice that ; the object is a traffic sign, so the ego vehicle should continue going ahead at the same speed. Secondly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed. Thirdly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 3 + ] + }, + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed, the reason is that there is no safety issue. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "No such action will lead to a collision.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Keep going at the same speed and brake suddenly.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is driving slowly.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + } + }, + "dfb8d8959b9944d69dcec6d05e419f04": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a golden sedan to the back of the ego vehicle, a black sedan car to the back of the ego vehicle, and a white truck to the back of the ego vehicle. The IDs of these objects are , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Going ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "A": "Firstly notice that , the object is turning left, so the ego vehicle should keep going ahead at the same speed. Secondly notice that , the object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that , it is stationary, so the ego vehicle should keep going ahead at the same speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 3 + ] + }, + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed because there is no safety issue. The probability of taking this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "No such action will lead to a collision.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Keep going at the same speed, accelerate, and proceed ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is driving fast.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + } + } + } + }, + "54cdaaae372d421fa4734d66f51a8c48": { + "key_frames": { + "542eaf1fc9b34895a9e55fab57cb4cf4": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a red car to the front of the ego vehicle, a white SUV to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Going ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to brake gently to a stop. The reason is to follow the traffic rules, and the probability is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Accelerate and go straight.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, brake suddenly, and slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is driving slowly.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + } + }, + "1b45a97a0e5e49fe9cd345dd4bd729c3": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white sedan car to the front of the ego vehicle, a blue sedan to the front right of the ego vehicle, a white sedan to the front right of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Turn right.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to remain stationary. The reason for this action is to follow the traffic rules. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Accelerating and going straight.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the right, and slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is not moving.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + } + }, + "d5e16062410f4e329d31a881b28e5c1c": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white truck to the front right of the ego vehicle, a white sedan to the front of the ego vehicle, a gray sedan to the front of the ego vehicle, a white sedan to the front right of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Going ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to remain stationary. The reason for taking this action is to follow traffic rules. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Turning right and accelerating.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is not moving.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + } + }, + "7abe79e5d3d745bcb237a4b7e03d1577": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white truck to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Going ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to remain stationary, the reason is to follow the traffic rules, high probability.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Moderate right turn.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the right.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is not moving.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + } + }, + "0dd14c14cea14dc1b0c0c9b0c9c7c4c3": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ?", + "A": "Going ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed, the reason being that there is no safety issue. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Accelerating and going straight can lead to a collision with .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle.", + "A": "The ego vehicle is going straight. The ego vehicle is driving with normal speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + } } } } diff --git a/challenge/test_eval.json b/challenge/test_eval.json new file mode 100644 index 0000000..6b23ba5 --- /dev/null +++ b/challenge/test_eval.json @@ -0,0 +1,982 @@ +{ + "f0f120e4d4b0441da90ec53b16ee169d": { + "key_frames": { + "4a0798f849ca477ab18009c3a20b7df2": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a brown SUV to the back of the ego vehicle, a black sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Stopped. C. Back up. D. Turn left.", + "A": "D", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "A": "Firstly notice that . The object is a traffic sign, so the ego vehicle should keep going ahead at the same speed. Secondly notice that . The object is turning left, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that . The object is going ahead, so the ego vehicle should keep going ahead at the same speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 3 + ] + }, + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed. The reason is to follow the traffic rules, which has a high probability.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "No such action will lead to a collision.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Keep going at the same speed, decelerate gradually without braking.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed.", + "A": "B", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + } + }, + "ffd1bdf020d145759224c629b501d2b2": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a champagne-colored sedan to the back of the ego vehicle, a green light to the front of the ego vehicle, and a red light to the front left of the ego vehicle. The IDs of these objects are , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + } + ], + "prediction": [ + { + "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "A": "Firstly, notice . The object is a traffic sign, so the ego vehicle should continue at the same speed. Secondly, notice . The object is a traffic sign, so the ego vehicle should accelerate and continue ahead. Thirdly, notice . The object is stationary, so the ego vehicle should continue ahead at the same speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 3 + ] + }, + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep stationary, the reason is to follow the traffic rules, high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Back up.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Keep going at the same speed, decelerate gradually without braking.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. B. The ego vehicle is steering to the left. The ego vehicle is driving fast. C. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving slowly.", + "A": "D", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + } + }, + "d9075c2a5f864a2b8abf41e703f4cf1c": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white sedan to the front left of the ego vehicle, a gray sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Going ahead. C. Turn right. D. Turn left.", + "A": "B", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "A": "Firstly notice that ; the object is a traffic sign, so the ego vehicle should continue going ahead at the same speed. Secondly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed. Thirdly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 3 + ] + }, + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed, the reason is that there is no safety issue. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "No such action will lead to a collision.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Keep going at the same speed and brake suddenly.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. B. The ego vehicle is going straight. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is not moving. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.", + "A": "B", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + } + }, + "dfb8d8959b9944d69dcec6d05e419f04": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a golden sedan to the back of the ego vehicle, a black sedan car to the back of the ego vehicle, and a white truck to the back of the ego vehicle. The IDs of these objects are , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Turn left. C. Going ahead. D. Turn right.", + "A": "C", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", + "A": "Firstly notice that , the object is turning left, so the ego vehicle should keep going ahead at the same speed. Secondly notice that , the object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that , it is stationary, so the ego vehicle should keep going ahead at the same speed.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 3 + ] + }, + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed because there is no safety issue. The probability of taking this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "No such action will lead to a collision.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Keep going at the same speed, accelerate, and proceed ahead.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. C. The ego vehicle is going straight. The ego vehicle is driving fast. D. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed.", + "A": "C", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + } + } + } + }, + "54cdaaae372d421fa4734d66f51a8c48": { + "key_frames": { + "542eaf1fc9b34895a9e55fab57cb4cf4": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a red car to the front of the ego vehicle, a white SUV to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Back up. C. Turn left. D. Going ahead.", + "A": "D", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to brake gently to a stop. The reason is to follow the traffic rules, and the probability is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Accelerate and go straight.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, brake suddenly, and slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed.", + "A": "C", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + } + }, + "1b45a97a0e5e49fe9cd345dd4bd729c3": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white sedan car to the front of the ego vehicle, a blue sedan to the front right of the ego vehicle, a white sedan to the front right of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Turn right. D. Stopped.", + "A": "C", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to remain stationary. The reason for this action is to follow the traffic rules. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Accelerating and going straight.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the right, and slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is steering to the right. The ego vehicle is driving slowly. D. The ego vehicle is steering to the right. The ego vehicle is driving fast.", + "A": "A", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + } + }, + "d5e16062410f4e329d31a881b28e5c1c": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white truck to the front right of the ego vehicle, a white sedan to the front of the ego vehicle, a gray sedan to the front of the ego vehicle, a white sedan to the front right of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Going ahead. D. Turn left.", + "A": "C", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to remain stationary. The reason for taking this action is to follow traffic rules. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Turning right and accelerating.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is steering to the right. The ego vehicle is driving very fast.", + "A": "A", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + } + }, + "7abe79e5d3d745bcb237a4b7e03d1577": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white truck to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Turn left. C. Stopped. D. Back up.", + "A": "A", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to remain stationary, the reason is to follow the traffic rules, high probability.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Moderate right turn.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the right.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is not moving. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed.", + "A": "B", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + } + }, + "0dd14c14cea14dc1b0c0c9b0c9c7c4c3": { + "QA": { + "perception": [ + { + "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", + "A": "There is a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , and .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 2 + ] + }, + { + "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Turn left. B. Turn right. C. Back up. D. Going ahead.", + "A": "D", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "prediction": [ + { + "Q": "Is a traffic sign or a road barrier?", + "A": "No.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ], + "planning": [ + { + "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", + "A": "The action is to keep going at the same speed, the reason being that there is no safety issue. The probability of this action is high.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "What actions taken by the ego vehicle can lead to a collision with ?", + "A": "Accelerating and going straight can lead to a collision with .", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + }, + { + "Q": "In this scenario, what are safe actions to take for the ego vehicle?", + "A": "Brake gently to a stop, slightly offset to the left.", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 1 + ] + } + ], + "behavior": [ + { + "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed.", + "A": "D", + "C": null, + "con_up": null, + "con_down": null, + "cluster": null, + "layer": null, + "tag": [ + 0 + ] + } + ] + }, + "image_paths": { + "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "CAM_BACK": "../nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + } + } + } + } +} \ No newline at end of file diff --git a/challenge/test_llama.json b/challenge/test_llama.json new file mode 100644 index 0000000..e4888ce --- /dev/null +++ b/challenge/test_llama.json @@ -0,0 +1,1388 @@ +[ + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a brown SUV to the back of the ego vehicle, a black sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and ." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Stopped. C. Back up. D. Turn left." + }, + { + "from": "gpt", + "value": "D" + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?" + }, + { + "from": "gpt", + "value": "Firstly notice that . The object is a traffic sign, so the ego vehicle should keep going ahead at the same speed. Secondly notice that . The object is turning left, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that . The object is going ahead, so the ego vehicle should keep going ahead at the same speed." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to keep going at the same speed. The reason is to follow the traffic rules, which has a high probability." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "No such action will lead to a collision." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Keep going at the same speed, decelerate gradually without braking." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_7", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed." + }, + { + "from": "gpt", + "value": "B" + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a champagne-colored sedan to the back of the ego vehicle, a green light to the front of the ego vehicle, and a red light to the front left of the ego vehicle. The IDs of these objects are , , and ." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?" + }, + { + "from": "gpt", + "value": "Firstly, notice . The object is a traffic sign, so the ego vehicle should continue at the same speed. Secondly, notice . The object is a traffic sign, so the ego vehicle should accelerate and continue ahead. Thirdly, notice . The object is stationary, so the ego vehicle should continue ahead at the same speed." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to keep stationary, the reason is to follow the traffic rules, high." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "Back up." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Keep going at the same speed, decelerate gradually without braking." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. B. The ego vehicle is steering to the left. The ego vehicle is driving fast. C. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving slowly." + }, + { + "from": "gpt", + "value": "D" + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a white sedan to the front left of the ego vehicle, a gray sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and ." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Going ahead. C. Turn right. D. Turn left." + }, + { + "from": "gpt", + "value": "B" + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?" + }, + { + "from": "gpt", + "value": "Firstly notice that ; the object is a traffic sign, so the ego vehicle should continue going ahead at the same speed. Secondly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed. Thirdly, notice that ; the object is going ahead, so the ego vehicle should continue going ahead at the same speed." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to keep going at the same speed, the reason is that there is no safety issue. The probability of this action is high." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "No such action will lead to a collision." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Keep going at the same speed and brake suddenly." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_d9075c2a5f864a2b8abf41e703f4cf1c_7", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291005162404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291005154799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291005170482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291005187558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291005197405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291005178113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. B. The ego vehicle is going straight. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is not moving. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed." + }, + { + "from": "gpt", + "value": "B" + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a golden sedan to the back of the ego vehicle, a black sedan car to the back of the ego vehicle, and a white truck to the back of the ego vehicle. The IDs of these objects are , , and ." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Turn left. C. Going ahead. D. Turn right." + }, + { + "from": "gpt", + "value": "C" + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?" + }, + { + "from": "gpt", + "value": "Firstly notice that , the object is turning left, so the ego vehicle should keep going ahead at the same speed. Secondly notice that , the object is going ahead, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that , it is stationary, so the ego vehicle should keep going ahead at the same speed." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to keep going at the same speed because there is no safety issue. The probability of taking this action is high." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "No such action will lead to a collision." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Keep going at the same speed, accelerate, and proceed ahead." + } + ] + }, + { + "id": "f0f120e4d4b0441da90ec53b16ee169d_dfb8d8959b9944d69dcec6d05e419f04_7", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291013612404.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291013604799.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291013620482.jpg", + "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291013637558.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291013647405.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291013628113.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the right. The ego vehicle is driving very fast. C. The ego vehicle is going straight. The ego vehicle is driving fast. D. The ego vehicle is steering to the left. The ego vehicle is driving with normal speed." + }, + { + "from": "gpt", + "value": "C" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a red car to the front of the ego vehicle, a white SUV to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and ." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Stopped. B. Back up. C. Turn left. D. Going ahead." + }, + { + "from": "gpt", + "value": "D" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to brake gently to a stop. The reason is to follow the traffic rules, and the probability is high." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "Accelerate and go straight." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Brake gently to a stop, brake suddenly, and slightly offset to the left." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_542eaf1fc9b34895a9e55fab57cb4cf4_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280039162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280039154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280039170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280039187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280039197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280039177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed." + }, + { + "from": "gpt", + "value": "C" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a white sedan car to the front of the ego vehicle, a blue sedan to the front right of the ego vehicle, a white sedan to the front right of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , , and ." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Turn right. D. Stopped." + }, + { + "from": "gpt", + "value": "C" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to remain stationary. The reason for this action is to follow the traffic rules. The probability of this action is high." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "Accelerating and going straight." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Brake gently to a stop, slightly offset to the right, and slightly offset to the left." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_1b45a97a0e5e49fe9cd345dd4bd729c3_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280042612460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280042604844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280042620339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280042637525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280042647423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280042627893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is steering to the right. The ego vehicle is driving slowly. D. The ego vehicle is steering to the right. The ego vehicle is driving fast." + }, + { + "from": "gpt", + "value": "A" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a white truck to the front right of the ego vehicle, a white sedan to the front of the ego vehicle, a gray sedan to the front of the ego vehicle, a white sedan to the front right of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , , and ." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Reverse parking. B. Back up. C. Going ahead. D. Turn left." + }, + { + "from": "gpt", + "value": "C" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to remain stationary. The reason for taking this action is to follow traffic rules. The probability of this action is high." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "Turning right and accelerating." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Brake gently to a stop, slightly offset to the left." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_d5e16062410f4e329d31a881b28e5c1c_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280045112460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280045104844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280045120339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280045137525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280045147423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280045127893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is not moving. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving slowly. C. The ego vehicle is steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is steering to the right. The ego vehicle is driving very fast." + }, + { + "from": "gpt", + "value": "A" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a white truck to the front of the ego vehicle, a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, and a red light to the front of the ego vehicle. The IDs of these objects are , , , and ." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Turn left. C. Stopped. D. Back up." + }, + { + "from": "gpt", + "value": "A" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to remain stationary, the reason is to follow the traffic rules, high probability." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "Moderate right turn." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Brake gently to a stop, slightly offset to the right." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_7abe79e5d3d745bcb237a4b7e03d1577_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280048162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280048154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280048170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280048187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280048197442.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280048177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is going straight. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is not moving. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving with normal speed." + }, + { + "from": "gpt", + "value": "B" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_0", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." + }, + { + "from": "gpt", + "value": "There is a white sedan to the front of the ego vehicle, a white van to the front of the ego vehicle, a black sedan to the front of the ego vehicle, and a no entry to the front of the ego vehicle. The IDs of these objects are , , , and ." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_1", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Turn left. B. Turn right. C. Back up. D. Going ahead." + }, + { + "from": "gpt", + "value": "D" + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_2", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIs a traffic sign or a road barrier?" + }, + { + "from": "gpt", + "value": "No." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_3", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" + }, + { + "from": "gpt", + "value": "The action is to keep going at the same speed, the reason being that there is no safety issue. The probability of this action is high." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_4", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" + }, + { + "from": "gpt", + "value": "Accelerating and going straight can lead to a collision with ." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_5", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" + }, + { + "from": "gpt", + "value": "Brake gently to a stop, slightly offset to the left." + } + ] + }, + { + "id": "54cdaaae372d421fa4734d66f51a8c48_0dd14c14cea14dc1b0c0c9b0c9c7c4c3_6", + "image": [ + "data/nuscenes/samples/CAM_FRONT/n015-2018-08-03-15-00-36+0800__CAM_FRONT__1533280036162460.jpg", + "data/nuscenes/samples/CAM_FRONT_LEFT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_LEFT__1533280036154844.jpg", + "data/nuscenes/samples/CAM_FRONT_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_FRONT_RIGHT__1533280036170339.jpg", + "data/nuscenes/samples/CAM_BACK/n015-2018-08-03-15-00-36+0800__CAM_BACK__1533280036187525.jpg", + "data/nuscenes/samples/CAM_BACK_LEFT/n015-2018-08-03-15-00-36+0800__CAM_BACK_LEFT__1533280036197423.jpg", + "data/nuscenes/samples/CAM_BACK_RIGHT/n015-2018-08-03-15-00-36+0800__CAM_BACK_RIGHT__1533280036177893.jpg" + ], + "conversations": [ + { + "from": "human", + "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the left. The ego vehicle is driving fast. B. The ego vehicle is steering to the left. The ego vehicle is driving very fast. C. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. D. The ego vehicle is going straight. The ego vehicle is driving with normal speed." + }, + { + "from": "gpt", + "value": "D" + } + ] + } +] \ No newline at end of file diff --git a/challenge/test_v1.json b/challenge/test_v1.json deleted file mode 100644 index 715318e..0000000 --- a/challenge/test_v1.json +++ /dev/null @@ -1,226 +0,0 @@ -{ - "f0f120e4d4b0441da90ec53b16ee169d": { - "key_frames": { - "4a0798f849ca477ab18009c3a20b7df2": { - "QA": { - "perception": [ - { - "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", - "A": "There is a brown SUV to the back of the ego vehicle, a black sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and .", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 2 - ] - }, - { - "Q": "What is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Turn right. C. Turn left. D. Stopped.", - "A": "C", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 0 - ] - } - ], - "prediction": [ - { - "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", - "A": "Firstly notice that . The object is a traffic sign, so the ego vehicle should keep going ahead at the same speed. Secondly notice that . The object is turning left, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that . The object is going ahead, so the ego vehicle should keep going ahead at the same speed.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 3 - ] - }, - { - "Q": "Is a traffic sign or a road barrier?", - "A": "No.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 0 - ] - } - ], - "planning": [ - { - "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", - "A": "The action is to keep going at the same speed. The reason is to follow the traffic rules, which has a high probability.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 1 - ] - }, - { - "Q": "What actions taken by the ego vehicle can lead to a collision with ?", - "A": "No such action will lead to a collision.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 1 - ] - }, - { - "Q": "In this scenario, what are safe actions to take for the ego vehicle?", - "A": "Keep going at the same speed, decelerate gradually without braking.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 1 - ] - } - ], - "behavior": [ - { - "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is steering to the right. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving fast.", - "A": "B", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 0 - ] - } - ] - }, - "image_paths": { - "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - } - }, - "ffd1bdf020d145759224c629b501d2b2": { - "QA": { - "perception": [ - { - "Q": "What are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision.", - "A": "There is a champagne-colored sedan to the back of the ego vehicle, a green light to the front of the ego vehicle, and a red light to the front left of the ego vehicle. The IDs of these objects are , , and .", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 2 - ] - } - ], - "prediction": [ - { - "Q": "What object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?", - "A": "Firstly, notice . The object is a traffic sign, so the ego vehicle should continue at the same speed. Secondly, notice . The object is a traffic sign, so the ego vehicle should accelerate and continue ahead. Thirdly, notice . The object is stationary, so the ego vehicle should continue ahead at the same speed.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 3 - ] - }, - { - "Q": "Is a traffic sign or a road barrier?", - "A": "No.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 0 - ] - } - ], - "planning": [ - { - "Q": "What actions could the ego vehicle take based on ? Why take this action and what's the probability?", - "A": "The action is to keep stationary, the reason is to follow the traffic rules, high.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 1 - ] - }, - { - "Q": "What actions taken by the ego vehicle can lead to a collision with ?", - "A": "Back up.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 1 - ] - }, - { - "Q": "In this scenario, what are safe actions to take for the ego vehicle?", - "A": "Keep going at the same speed, decelerate gradually without braking.", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 1 - ] - } - ], - "behavior": [ - { - "Q": "Predict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is steering to the left. The ego vehicle is driving fast. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is going straight. The ego vehicle is driving fast.", - "A": "C", - "C": null, - "con_up": null, - "con_down": null, - "cluster": null, - "layer": null, - "tag": [ - 0 - ] - } - ] - }, - "image_paths": { - "CAM_FRONT": "../nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "CAM_FRONT_LEFT": "../nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "CAM_FRONT_RIGHT": "../nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "CAM_BACK": "../nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "CAM_BACK_LEFT": "../nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "CAM_BACK_RIGHT": "../nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - } - } - } - } -} \ No newline at end of file diff --git a/challenge/test_v2.json b/challenge/test_v2.json deleted file mode 100644 index d906f27..0000000 --- a/challenge/test_v2.json +++ /dev/null @@ -1,317 +0,0 @@ -[ - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_0", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." - }, - { - "from": "gpt", - "value": "There is a brown SUV to the back of the ego vehicle, a black sedan to the back of the ego vehicle, and a green light to the front of the ego vehicle. The IDs of these objects are , , and ." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_1", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat is the moving status of object ? Please select the correct answer from the following options: A. Going ahead. B. Turn right. C. Turn left. D. Stopped." - }, - { - "from": "gpt", - "value": "C" - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_2", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?" - }, - { - "from": "gpt", - "value": "Firstly notice that . The object is a traffic sign, so the ego vehicle should keep going ahead at the same speed. Secondly notice that . The object is turning left, so the ego vehicle should keep going ahead at the same speed. Thirdly notice that . The object is going ahead, so the ego vehicle should keep going ahead at the same speed." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_3", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nIs a traffic sign or a road barrier?" - }, - { - "from": "gpt", - "value": "No." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_4", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" - }, - { - "from": "gpt", - "value": "The action is to keep going at the same speed. The reason is to follow the traffic rules, which has a high probability." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_5", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" - }, - { - "from": "gpt", - "value": "No such action will lead to a collision." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_6", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" - }, - { - "from": "gpt", - "value": "Keep going at the same speed, decelerate gradually without braking." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_4a0798f849ca477ab18009c3a20b7df2_7", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291010612404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291010604799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291010620482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291010637558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291010647405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291010628113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is slightly steering to the right. The ego vehicle is driving slowly. B. The ego vehicle is going straight. The ego vehicle is driving fast. C. The ego vehicle is steering to the right. The ego vehicle is driving very fast. D. The ego vehicle is slightly steering to the right. The ego vehicle is driving fast." - }, - { - "from": "gpt", - "value": "B" - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_0", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat are the important objects in the current scene? Those objects will be considered for the future reasoning and driving decision." - }, - { - "from": "gpt", - "value": "There is a champagne-colored sedan to the back of the ego vehicle, a green light to the front of the ego vehicle, and a red light to the front left of the ego vehicle. The IDs of these objects are , , and ." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_1", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat object should the ego vehicle notice first when the ego vehicle is getting to the next possible location? What is the state of the object that is first noticed by the ego vehicle and what action should the ego vehicle take? What object should the ego vehicle notice second when the ego vehicle is getting to the next possible location? What is the state of the object perceived by the ego vehicle as second and what action should the ego vehicle take? What object should the ego vehicle notice third? What is the state of the object perceived by the ego vehicle as third and what action should the ego vehicle take?" - }, - { - "from": "gpt", - "value": "Firstly, notice . The object is a traffic sign, so the ego vehicle should continue at the same speed. Secondly, notice . The object is a traffic sign, so the ego vehicle should accelerate and continue ahead. Thirdly, notice . The object is stationary, so the ego vehicle should continue ahead at the same speed." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_2", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nIs a traffic sign or a road barrier?" - }, - { - "from": "gpt", - "value": "No." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_3", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat actions could the ego vehicle take based on ? Why take this action and what's the probability?" - }, - { - "from": "gpt", - "value": "The action is to keep stationary, the reason is to follow the traffic rules, high." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_4", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nWhat actions taken by the ego vehicle can lead to a collision with ?" - }, - { - "from": "gpt", - "value": "Back up." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_5", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nIn this scenario, what are safe actions to take for the ego vehicle?" - }, - { - "from": "gpt", - "value": "Keep going at the same speed, decelerate gradually without braking." - } - ] - }, - { - "id": "f0f120e4d4b0441da90ec53b16ee169d_ffd1bdf020d145759224c629b501d2b2_6", - "image": [ - "data/nuscenes/samples/CAM_FRONT/n008-2018-09-18-13-10-39-0400__CAM_FRONT__1537291002262404.jpg", - "data/nuscenes/samples/CAM_FRONT_LEFT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_LEFT__1537291002254799.jpg", - "data/nuscenes/samples/CAM_FRONT_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_FRONT_RIGHT__1537291002270482.jpg", - "data/nuscenes/samples/CAM_BACK/n008-2018-09-18-13-10-39-0400__CAM_BACK__1537291002287558.jpg", - "data/nuscenes/samples/CAM_BACK_LEFT/n008-2018-09-18-13-10-39-0400__CAM_BACK_LEFT__1537291002297405.jpg", - "data/nuscenes/samples/CAM_BACK_RIGHT/n008-2018-09-18-13-10-39-0400__CAM_BACK_RIGHT__1537291002278113.jpg" - ], - "conversations": [ - { - "from": "human", - "value": "\nPredict the behavior of the ego vehicle. Please select the correct answer from the following options: A. The ego vehicle is steering to the left. The ego vehicle is driving fast. B. The ego vehicle is slightly steering to the left. The ego vehicle is driving with normal speed. C. The ego vehicle is going straight. The ego vehicle is driving slowly. D. The ego vehicle is going straight. The ego vehicle is driving fast." - }, - { - "from": "gpt", - "value": "C" - } - ] - } -] \ No newline at end of file From c03a44e43e5d5bdb9b1821c42cb6cbbac64eddb4 Mon Sep 17 00:00:00 2001 From: Linyan Huang Date: Tue, 5 Mar 2024 09:32:58 +0000 Subject: [PATCH 2/6] Update sample data --- challenge/{llama-adapter-DriveLM.json => output.json} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename challenge/{llama-adapter-DriveLM.json => output.json} (100%) diff --git a/challenge/llama-adapter-DriveLM.json b/challenge/output.json similarity index 100% rename from challenge/llama-adapter-DriveLM.json rename to challenge/output.json From c4444433c271fabcab1ef8d08829c2d9d85d8089 Mon Sep 17 00:00:00 2001 From: Linyan Huang Date: Tue, 5 Mar 2024 09:40:34 +0000 Subject: [PATCH 3/6] minor fix --- challenge/convert2llama.py | 4 ++-- challenge/convert_data.py | 2 +- challenge/evaluation.py | 4 ++-- challenge/llama_adapter_v2_multimodal7b/demo.py | 4 ++-- .../llama_adapter_v2_multimodal7b/finetune_data_config.yaml | 2 +- 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/challenge/convert2llama.py b/challenge/convert2llama.py index 3822e32..84e2526 100644 --- a/challenge/convert2llama.py +++ b/challenge/convert2llama.py @@ -42,6 +42,6 @@ def convert2llama(root, dst): if __name__ == '__main__': - root = "test_v1.json" - dst = "test_v2.json" + root = "test_eval.json" + dst = "test_llama.json" convert2llama(root, dst) diff --git a/challenge/convert_data.py b/challenge/convert_data.py index 2669ff5..1999d62 100644 --- a/challenge/convert_data.py +++ b/challenge/convert_data.py @@ -71,5 +71,5 @@ def loop_test(root, dst): if __name__ == '__main__': root = "test.json" - dst = "test_v1.json" + dst = "test_eval.json" loop_test(root, dst) diff --git a/challenge/evaluation.py b/challenge/evaluation.py index 38de7a6..1b8a06e 100644 --- a/challenge/evaluation.py +++ b/challenge/evaluation.py @@ -97,8 +97,8 @@ def forward(self, tag, answer, GT): if __name__ == '__main__': # get args parser = argparse.ArgumentParser(description='Evaluation') - parser.add_argument('--root_path1', type=str, default="./llama-adapter-DriveLM.json", help='path to prediction file') - parser.add_argument('--root_path2', type=str, default="./test_v1.json", help='path to test file') + parser.add_argument('--root_path1', type=str, default="./output.json", help='path to prediction file') + parser.add_argument('--root_path2', type=str, default="./test_eval.json", help='path to test file') args = parser.parse_args() with open(args.root_path1, 'r') as f :#, \ diff --git a/challenge/llama_adapter_v2_multimodal7b/demo.py b/challenge/llama_adapter_v2_multimodal7b/demo.py index 94817f5..f1b4852 100644 --- a/challenge/llama_adapter_v2_multimodal7b/demo.py +++ b/challenge/llama_adapter_v2_multimodal7b/demo.py @@ -16,8 +16,8 @@ parser = argparse.ArgumentParser(description='LLAMA Adapter') parser.add_argument('--llama_dir', type=str, default="/path/to/llama_model_weights", help='path to llama model weights') parser.add_argument('--checkpoint', type=str, default="/path/to/pre-trained/checkpoint.pth", help='path to pre-trained checkpoint') -parser.add_argument('--data', type=str, default="../test_v2.json", help='path to test data') -parser.add_argument('--output', type=str, default="../llama-adapter-DriveLM.json", help='path to output file') +parser.add_argument('--data', type=str, default="../test_llama.json", help='path to test data') +parser.add_argument('--output', type=str, default="../output.json", help='path to output file') args = parser.parse_args() device = "cuda" if torch.cuda.is_available() else "cpu" diff --git a/challenge/llama_adapter_v2_multimodal7b/finetune_data_config.yaml b/challenge/llama_adapter_v2_multimodal7b/finetune_data_config.yaml index 046456c..a568a29 100644 --- a/challenge/llama_adapter_v2_multimodal7b/finetune_data_config.yaml +++ b/challenge/llama_adapter_v2_multimodal7b/finetune_data_config.yaml @@ -1,4 +1,4 @@ META: - - 'v1_0_train_nus_llama.json' + - 'test_llama.json' From 2339c963e8fb9e12943442b5c69fed1658e4450e Mon Sep 17 00:00:00 2001 From: Linyan Huang Date: Tue, 5 Mar 2024 09:40:43 +0000 Subject: [PATCH 4/6] Update README.md --- challenge/README.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/challenge/README.md b/challenge/README.md index 98ea080..456b85c 100644 --- a/challenge/README.md +++ b/challenge/README.md @@ -73,17 +73,17 @@ Transform the obtained test.json data into the required test format. # make sure you are under ./challenge python convert_data.py ``` -Then we will get the test_v1.json in challenge folder. The example of test_v1.json can be found in [test_v1.json](test_v1.json) +Then we will get the test_eval.json in challenge folder. The example of test_eval.json can be found in [test_eval.json](test_eval.json) We use llama-adapter v2 as our baseline. If you want to convert data into llama-adapter format: ```bash -# The following script assumes that you prepare the test_v1.json under ./challenge +# The following script assumes that you prepare the test_eval.json under ./challenge # make sure you are under ./challenge python convert2llama.py ``` -Then we will get the test_v2.json in challenge folder. The example of test_v2.json can be found in [test_v2.json](test_v2.json) +Then we will get the test_llama.json in challenge folder. The example of test_llama.json can be found in [test_llama.json](test_llama.json) -[test_v1.json](test_v1.json) is used for evaluation. [test_v2.json](test_v2.json) is used for training and inference of the baseline. +[test_eval.json](test_eval.json) is used for evaluation. [test_llama.json](test_llama.json) is used for training and inference of the baseline. ## How to run baseline @@ -111,7 +111,7 @@ pip install -r requirements.txt ### Train baseline You should modify the [finetune_data_config.yaml](llama_adapter_v2_multimodal7b/finetune_data_config.yaml#L2) to specify the datasets for fine-tuning. -The format of datasets refers to [test_v2.json](test_v2.json). +The format of datasets refers to [test_llama.json](test_llama.json). The pre-trained checkpoint can be downloaded in [ckpts](https://github.com/OpenGVLab/LLaMA-Adapter/releases/tag/v.2.0.0). @@ -129,9 +129,9 @@ finetune_data_config.yaml /output/path ```bash # /path/to/llama_model_weights and /path/to/pre-trained/checkpoint.pth need to be modified by your path # make sure you are under ./challenge/llama_adapter_v2_multimodal7b -python demo.py --llama_dir /path/to/llama_model_weights --checkpoint /path/to/pre-trained/checkpoint.pth --data ../test_v2.json --output ../llama-adapter-DriveLM.json +python demo.py --llama_dir /path/to/llama_model_weights --checkpoint /path/to/pre-trained/checkpoint.pth --data ../test_llama.json --output ../output.json ``` -Then we will get the [llama-adapter-DriveLM.json](llama-adapter-DriveLM.json), which are the predicted answers used for evaluation purposes. +Then we will get the [output.json](output.json), which are the predicted answers used for evaluation purposes. ## How to Eval @@ -165,16 +165,16 @@ python -c "import language_evaluation; language_evaluation.download('coco')" We have implemented three types of evaluation methods: Accuracy, ChatGPT Score, Language Evaluation and Match Score. The [final score](evaluation.py#L157) is the weighted average of four metrics. -The inputs required for evaluation are [llama-adapter-DriveLM.json](llama-adapter-DriveLM.json) and [test_v1.json](test_v1.json). +The inputs required for evaluation are [output.json](output.json) and [test_eval.json](test_eval.json). -1. Replace [root_path1](evaluation.py#L97) with the path of your models' output. The example of models' output can be found in [output](llama-adapter-DriveLM.json). -2. Replace [root_path2](evaluation.py#L101) with the path of test_v1.json. The example of test_v1.json can be found in [test_v1.json](test_v1.json) +1. Replace [root_path1](evaluation.py#L97) with the path of your models' output. The example of models' output can be found in [output](output.json). +2. Replace [root_path2](evaluation.py#L101) with the path of test_eval.json. The example of test_eval.json can be found in [test_eval.json](test_eval.json) 3. Replace [API-KEY](chatgpt.py#L17) with your own chatGPT api key. ```bash -# The following script assumes that you prepare the llama-adapter-DriveLM.json and test_v1.json under ./challenge +# The following script assumes that you prepare the output.json and test_eval.json under ./challenge # make sure you are under ./challenge -python evaluation.py --root_path1 ./llama-adapter-DriveLM.json --root_path2 ./test_v1.json +python evaluation.py --root_path1 ./output.json --root_path2 ./test_eval.json ``` ### Results From 2b55db17424a1cdc05c2fcac1b1cfae33ed388a6 Mon Sep 17 00:00:00 2001 From: Linyan Huang Date: Tue, 5 Mar 2024 09:43:04 +0000 Subject: [PATCH 5/6] Update README.md --- challenge/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/challenge/README.md b/challenge/README.md index 456b85c..5595a37 100644 --- a/challenge/README.md +++ b/challenge/README.md @@ -113,7 +113,7 @@ pip install -r requirements.txt You should modify the [finetune_data_config.yaml](llama_adapter_v2_multimodal7b/finetune_data_config.yaml#L2) to specify the datasets for fine-tuning. The format of datasets refers to [test_llama.json](test_llama.json). -The pre-trained checkpoint can be downloaded in [ckpts](https://github.com/OpenGVLab/LLaMA-Adapter/releases/tag/v.2.0.0). +The pre-trained checkpoint can be downloaded in [ckpts](https://github.com/OpenGVLab/LLaMA-Adapter/releases/tag/v.2.0.0). You can choose any one of them. Then we can train baseline as follows. ```bash From e74eca87ffc061f2e7fd794887a692a42552fd0b Mon Sep 17 00:00:00 2001 From: Linyan Huang Date: Tue, 5 Mar 2024 09:48:08 +0000 Subject: [PATCH 6/6] Update README.md --- challenge/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/challenge/README.md b/challenge/README.md index 5595a37..00721ab 100644 --- a/challenge/README.md +++ b/challenge/README.md @@ -181,10 +181,10 @@ python evaluation.py --root_path1 ./output.json --root_path2 ./test_eval.json The zero-shot results of baseline on the sampled data are as follows: ``` accuracy: 0.0 -chatgpt: 78.5 -match: 23.75 -language score: {'val/Bleu_1': 0.029183757177883535, 'val/Bleu_2': 0.00017003737042789148, 'val/Bleu_3': 3.066026234534233e-05, 'val/Bleu_4': 1.3024512157157705e-05, 'val/ROUGE_L': 0.05928706665796174, 'val/CIDEr': 0.05818698178494484} -final score: 0.36633034231114403 +chatgpt: 65.11111111111111 +match: 28.25 +language score: {'val/Bleu_1': 0.0495223110147729, 'val/Bleu_2': 0.00021977465683011536, 'val/Bleu_3': 3.6312541763196866e-05, 'val/Bleu_4': 1.4776149283286042e-05, 'val/ROUGE_L': 0.08383567940883102, 'val/CIDEr': 0.09901486412073952} +final score: 0.3240234750718823 ```