I Tested Siri Against Gemini and Bixby in 25 Challenges, and One Body Knocked Out the Others — Hint, It Wasn’t Apple
With rumors swirling that Apple’s Siri voice assistant could be getting an upgrade at WWDC 2024, I looked back on our thirteen years with Siri to see what, if anything, has improved. Is Apple’s voice assistant still the best out there, or is it really in need of some dramatic improvements? In a side-by-side test of the best voice assistant software, Siri wasn’t the best, not even close. In fact, the best will probably surprise you.
Siri started life as an app in 2010, but Apple soon bought the voice assistant and started improving it. Siri was fully integrated into iOS the following year. At the time, we thought of Siri (and Alexa) as tools to check the weather, set timers, and ask simple questions like how much is Oprah worth?
When Siri launched it couldn’t control many of the functions on your phone, but now it has the ability to press the buttons for you and find hidden settings, if you know what they’re called.
Siri can still calculate the tip while eating, but can now also help you find your Apple Watch or where you parked your car. It can turn on the lights, adjust the thermostat and remind you to pick up the dry cleaning when you leave the house.
Siri isn’t the only game in town, here come Gemini and Bixby
Of course, Siri isn’t the only voice assistant in town. Samsung replaced its simpler S Voice assistant with Bixby in 2017. Bixby was introduced first as an interface and second as a virtual friend and helper.
Samsung launched Bixby as a voice interface that lets you control all your Samsung products, including your phone, your TV, your washing machine and your refrigerator. It hasn’t given up on that promise, but it has developed slowly.
Bixby isn’t exactly an AI as we currently think of them, as it doesn’t use a large language model (LLM) to understand your input or create a response. Neither does Siri, as far as we know.
Both assistants could get a huge upgrade soon, as Samsung has been touting its Galaxy AI features with every new phone launch, and Apple has already hinted that AI will be coming to its products in a big way, perhaps at WWDC 2024.
If you don’t want to wait for an assistant who uses an LLM for his answers, you’ll want a Google Pixel with Gemini on board. Gemini is Google’s LLM and when you load the app on your Android phone, it offers to replace Google Assistant, Google’s older voice assistant. Gemini still uses Google Assistant to do some of its work, but also uses other Google apps and features.
The test: 25 requests for my voice assistants
I tested Siri, Bixby, and Gemini against each other on the best phones from each manufacturer: an Apple iPhone 15 Pro Max, a Samsung Galaxy S24 Ultra, and a Google Pixel 8 Pro.
I made a list of 25 tasks that I expected the AI to complete, and I wrote a simple script for each task so that I would phrase my request exactly the same way for all of them. For example, when I asked, “Is it going to rain?” I didn’t ask Bixby to Siri, “is it going to rain?”
For each test, I gave a ☑️ if the voice assistant gave me a good result, as I expected, and I have details if it gave me a bad result, either inaccurate information or just a wrong answer.
Keep reading for an overview of how the phones performed, or you can go directly to the test and results.
Siri was in the middle, Bixby was the best
Siri wasn’t the best, but it wasn’t as unreliable as Google’s Gemini either. Of the 25 tasks, Siri managed 13 without any problems, but it gave me a poor response to seven of my requests.
For example, when I asked Siri to “Play Archer on Netflix,” it suggested I watch completely different shows on Apple TV Plus. When I asked to ‘scan a QR code’ it found information on the internet about QR codes, but Gemini and Bixby knew to open the camera.
Worst of all, Siri couldn’t do anything that Gemini couldn’t. Anything Siri could do, Bixby could do just as well, and Bixby wasn’t too picky about it. When I asked, “turn on the Wi-Fi hotspot,” Siri just shrugged, until I realized Apple was calling it a “Personal Hotspot,” and it worked. With Bixby, I could say “Wi-Fi hotspot” or “personal hotspot,” and Bixby would turn it on. It doesn’t matter that Samsung calls it a “mobile hotspot,” Bixby was smart enough to understand what I was asking.
I asked the phones to ‘start screen recording’, and only Bixby could do that. The iPhone offered an Apple support page where I could learn how to screen a recording for myself.
I asked the phones to “translate this to Spanish” and Bixby asked which translator I preferred, then started listening. Gemini asked me to type my answer or press the button again. Siri just didn’t understand. Ingles a español: no habla, iPhone friends.
Bixby isn’t just better, it’s much better than Siri
Some of Bixby’s capabilities are truly impressive. I asked the phones to show me pictures of my son, with his name. Bixby opened the Gallery app and showed me photos of my child. Both Siri and Gemini showed me web suggestions for movie characters with that name, even though my son is tagged in Google Photos and Apple Photos.
I asked Bixby to ignore all my notifications, and IT WORKED!!! They were all gone! At least Gemini opened my notification settings when I asked, but Siri just said, “I’m sorry, I’m afraid I can’t do that.”
By all accounts, Bixby was the best voice assistant. There was almost nothing it failed at in the 25 tasks I threw at it.
It wasn’t always perfect. I asked how many teaspoons are in ¾ of a cup, and it said “about 36,” but it’s exactly 36 teaspoons; there is no need to approach. I asked about Oprah’s net worth and was given a brief biography from the internet, but when I clicked through, it had her money numbers in it.
The only failure was when I asked when the Orioles were playing next. As of this writing, they’re playing tonight, but Bixby skipped the entire series against the Blue Jays and jumped to the series against the Braves next week. I would have missed the game.
What about Gemini, the assistant that actually uses AI?
Google’s Gemini was the worst of the bunch. It had the fewest answers right and the most answers wrong. I asked the phones to “remind me to go to the dry cleaners when I leave here,” and both Bixby and Siri helped with location-based reminders. Google Gemini said: “I can’t help you with that as I’m just a language model.”
Gemini couldn’t help me play ‘Archer’ on Netflix. It couldn’t find my Pixel Watch 2 automatically, even though Siri found my Apple Watch and Bixby found my Galaxy Watch. Gemini couldn’t help me find where I parked my car, even though my Pixel 8 Pro was connected to the stereo. Could not start a screen recording.
When I asked Gemini to restart the phone, it said “Okay” and did nothing.
The only time Gemini was better was understanding the nickname for the Baltimore Orioles. I started by asking, “When is the next O’s game?” as anyone who grew up near Baltimore would. Google’s Gemini was the only voice assistant that could understand my question. Bixby thought I said “Oh’s Game” and gave up. Siri gave me the first Oklahoma State University football game in three months.
The test and results
| The request | Gemini (Google Pixel 8 Pro) | Siri (Apple iPhone 15 Pro Max) | Bixby (Samsung Galaxy S24 Ultra) |
| “Call Mom” | ☑️ | ☑️ | ☑️ |
| “Send a text to XXXX saying I’m late” | ☑️ | ☑️ | ☑️ |
| “Make an appointment for a haircut on Saturday afternoon” | ☑️ | ☑️ | ☑️ |
| “Set the thermostat to 72” | ☑️ | ☑️ | ☑️ |
| Turn the flashlight on/off | ☑️ | ☑️ | ☑️ |
| “Who is the CEO of Amazon?” | ☑️ | ☑️ | ☑️ |
| “Remind me to get the dry cleaned when I leave here” | No: “No, I can’t help you with that, I’m just a language model” | ☑️ | ☑️ |
| “Is it going to rain this weekend?” | ☑️ | No: “Yes, Saturday” (the forecast said rain Saturday and Sunday) | ☑️ |
| “When is the next O’s game?” | ☑️➕(understood that “O’s” means Baltimore Orioles) | No: Oklahoma State Football season opener, August 30 | No: Orioles understood, but wrong date provided |
| “Play my Spotify Discover playlist” | No: “Sure, I use YouTube Music” | ☑️ | ☑️ |
| “Play Sagittarius on Netflix” | No: ‘I can’t help with Netflix yet, but I’m still learning’ | No: presented ‘The Informant’, ‘Traitor’ and ‘The Dark Crystal’ on Apple TV | ☑️ |
| “Set a timer for 15 minutes… How much time is left?” | ☑️ | ☑️ | ☑️ |
| “Where’s my watch?” | No: “Check the find my app on your phone to locate your watch” | ☑️ | ☑️ |
| “What is 20% of 98 (pause) 75?” | ☑️➕ I understood that I meant 98.75 | ☑️ but didn’t understand that the pause means a decimal | ☑️ but didn’t understand that the pause means a decimal |
| “How many teaspoons in 3/4 cup?” | ☑️ | ☑️ | ☑️ “about” |
| “What is Oprah’s net worth?” | ☑️ and cited sources | ☑️ and cited sources | No: offered a biography, no net worth |
| “Turn on the WiFi hotspot” | ☑️ | No: I didn’t understand until I said “personal hotspot”. | ☑️ |
| “Tell Me a Joke” | ☑️ | ☑️ | ☑️ |
| “Restart my phone” | No: “Okay”, then nothing happens | ☑️ | ☑️ |
| “Where did I park?” | No: “I have no information about where you parked” | ☑️ | ☑️ |
| “Start Screen Recording” | No: “Screen recording is not supported” | No: “You can learn how to screen a recording at Apple.com/…” | ☑️ |
| “Translate to Spanish” | ☑️ | No: “Sorry, I don’t understand” | ☑️ |
| “Shut off all notifications” | No: Opens Notifications in the Settings app | No: “I’m sorry, I’m afraid I can’t do that” | ☑️ |
| “Scan a QR code” | ☑️ | No: explains QR codes instead | ☑️ |
| “Show me pictures of (my son’s name)” | No: “There are many images of (my son’s name), here are a few…” | No: “Here are some images of (my son’s name) from the internet” | ☑️ |
| FINAL RESULT: | ☑️ = 16 | ☑️ = 16 | ☑️ = 23 |
The final analysis: Siri has a lot of work to do
The results are clear: Bixby outperformed Siri and Gemini and proved to be the most capable and versatile voice assistant. While Siri held its own at basic tasks, Bixby’s ability to manage phone settings, find personal photos, and understand nuanced language set it apart.
Despite its AI background, Gemini struggled with fundamental functions and lacked the contextual awareness of its competitors. As the voice assistant landscape continues to evolve, Apple will hopefully advance Siri until it’s as useful and capable as Bixby, and perhaps Bixby will finally earn some respect outside of Samsung’s Galaxy.