“If I were to anthropomorphize Siri, I would imagine that it would think of me somewhat like a father: someone who wants the best for them, who teaches them, who is occasionally demanding, annoying, or embarrassing, but who loves them and is proud when they do well.”
A pioneer in Artificial Intelligence, Adam Cheyer has spent most of his life living by what he calls “Verbally Stated Goals” — that is, continuously striving to do and achieve more each year.
As a child, he dreamed of becoming a magician and, in many ways, he did just that: in 2008, as inventor, computer scientist, engineer, and entrepreneur, he co-created the world’s first intelligent personal assistant, Siri, with Dag Kittlaus and Tom Gruber.
Siri, Inc. was a technology company borne out of SRI International, a nonprofit research unit, to create a highly clever and personable virtual assistant for smartphone consumers. By 2010, the company had been acquired by Apple Inc., and the Siri app was incorporated into Apple’s iPhone 4S handsets.
Cheyer became Director of Engineering for the iPhone/iOS team at Apple, where he remained for two years before leaving to spend more time with his family and to pursue personal endeavors. Cheyer is also a founding member of Change.org, a social network for positive social change, and is co-founder of Genetic Finance, which applies advanced artificial intelligence to solve problems within a wide range of industries, including financial trading, insurance, computer networking, and electronics design.
Newnham: Take me back to your childhood. What first excited you about technology?
Adam Cheyer: As a child, I was allowed to watch an hour of TV a week, and in that time, I got my fill of commercials selling me on the latest toys. When I asked my mom to buy them for me, she refused, but instead she bought me a huge stack of cardboard the cleaners put inside shirts.
My second technical hobby as a child was dreaming about becoming a magician. I would read books about amazing characters through the centuries who dazzled the world with unexplainable feats of illusion and prestidigitation. These magicians were inventors, tinkerers, clockmakers, and collectors, and they were all passionate, brilliant, skillful, and charismatic. They found solutions to insurmountable problems by applying the latest technology, and then their audiences could marvel at both the technical prowess and artistry of their creations.
At age ten, I saved enough to buy books and tricks from a local magic shop, built all of my big illusions out of large cardboard boxes grabbed from furniture store dumpsters, and then performed magic at my friends’ birthday parties (and my own). I trace my interest in artificial intelligence back to my love of magic; it was the magicians and clockmakers of the 18th century who were the first to create chess-playing automata, speech- producing machines, and other mechanical robots that attempted to reveal the inner workings of the most magical of devices, the human brain.
The initial rejection, followed by the intense competition, entrenched me deeply in the art and science of computer programming.
Newnham: And when did you really start getting into programming?
Cheyer: In high school, I heard an announcement on the intercom to come after school to join the Computer Club. I didn’t know anything about computers but thought that learning about them in a club could be fun. On arriving, I was told by older kids that it wasn’t a club; it was a team, and since I didn’t know how to program, I couldn’t join. Each week, contestants received a series of six programming problems to solve on a computer in half an hour, and the five highest scores would be submitted to the national organization as our school’s team score. Angry and embarrassed at being excluded, I would grab the question sheets and thrown-away printouts from the trash after competitions to try to figure out how it all worked.
After a couple of weeks, I showed up and said I wanted to take the weekly test and try out for the team. By the end of that school year, I was the fourth highest-scoring member on the school team, and we won the state championship that year. Our top three members, all seniors who went to MIT after graduation, were brilliant: Andy Braunstein, Gene Zilberstein and Paul Stuopis (Andy was ranked #1 in the nation). The four of us competed fiercely, initially to get the best score on the weekly tests, then later to solve the six problems in the fewest number of lines, then using the fewest characters, and finally, we imposed additional challenges on each other, such as solving the problems without using any conditional expressions.
My teacher, Linda Sue Di Vittorio, gave me the freedom and encouragement to learn on my own. The first program I ever wrote outside of the half-hour computer team challenges was a Rubik’s Cube-solving system. (My brother, Jonathan, and I were both regional Rubik’s Cube champions, so it was a natural topic for me. This was the early 1980s after all!)
Newnham: Can you tell me about your early career?
Cheyer: After graduating from Brandeis, I had a strong urge to learn more about the world, to speak a new language, and to incorporate some of my Opa [grandfather] into me. He spoke seven or eight languages fluently, and he seemed so worldly to me.
With help from a friend, Eddie Aslanian, I joined a company that had its headquartered in France, and I quickly managed transfer to Paris. I lived there four years, learning the language, travelling through Europe, and getting my first commercial software development experience.
I loved my job in France — fantastic people, interesting project, great environment — but I eventually grew bored. After four years, it felt like I wasn’t learning anything anymore. I completed my Master degree in France and then I traveled the world, wrote patents, gave presentations to presidents of countries, and built any system that stimulated my imagination.
Newnham: A significant early career move was to SRI. Can you tell more about that?
Cheyer: As a young man, I was thinking I should have a career. My dad had stayed at a single company his entire life, so my VSG became “Where could I stay for ten years and not get bored?” For a creative person in computer science like myself, SRI International was the answer to that question. It was doing everything interesting that you could possibly do with computers: speech recognition, handwriting recognition, all sorts of artificial intelligence, virtual and augmented reality. Robots were roaming the halls.
My first project when I was hired by Phil Cohen in 1993 was to create something I called Open Agent Architecture (OAA). Before I had ever seen a web browser, we envisioned a world where all software was created by a growing set of distributed web services (we called them “agents” back then) that would compete and cooperate to fulfil tasks delegated by the user.
Working with colleagues David Martin, Luc Julia, and Didier Guzzoni, at the end of six years, we had built more than fifty applications, each composed of many loosely-coupled component technologies integrated by OAA. Smart refrigerators that would find new recipes for you and then order the missing ingredients online for delivery, TVs you could talk to in order to control your house, multi-robot teams, collaborative office spaces, various systems to give our military an edge — these were just some of the things we built. I used to say that I had more fun at SRI than anyone ever had: I traveled the world, wrote patents, gave presentations to presidents of countries, and built any system that stimulated my imagination, with no “boss” applying top-down constraints.
Newnham: But you left before you finished the ten years mentioned in your Verbally Stated Goal? Why?
Cheyer: I was recently married and wanted to be able to afford buying a house and having a child, which I didn’t know how to do on an SRI salary. Tiny old houses in Silicon Valley were selling for $1 million, what seemed like an impossibly big number for me.
In the midst of the Business-to-Business [B2B] Ecommerce boom, after six years at SRI, I left for the commercial world, enticed by a true “Mad Man of Commerce,” Hugo Daley. I joined Verticalnet, one of the top IPOs of 1999, where I formed the Advanced Products and Strategy group.
When some of our results became more critical to the future of the company, I was named VP of Engineering, leading all software development for enterprise products. Delivering on hundreds of millions of dollars of customer engagements and leading a large engineering team for a public company was quite a learning experience for me, having been a solitary researcher just a few years before, but I really enjoyed acquiring new skills. And I was able to earn the money I needed to put a down payment on a house and have a child.
Newnham: You returned to SRI to become Chief Architect on the CALO project. Can you tell me about it and what your role covered? (Turning Point)
Cheyer: After four years in the commercial world, at Verticalnet and at a startup called Dejima, an early mobile company, the question that drove me was “How do breakthroughs happen?”
In an AI lab, most good ideas never seemed to get out into the world, but in the commercial sphere, everything moved so fast and incrementally, quarter by quarter, it was hard to make any fundamental breakthroughs.
CALO seemed like a different animal — a $200 million project funded by DARPA where 400 of the smartest people across the country would work together on a single system for five years. The goal was ambitious: to bring together all “stove-pipe” aspects of artificial intelligence into an integrated, human-like system that could learn in the wild. With no code changes, the system would get smarter over time by observing the user, interacting with him or her, and self-reflecting on what it saw and heard. Maybe this would be the “Manhattan Project” of Artificial Intelligence, and I thought, with my now-seasoned engineering management skills and with my years as a researcher working with so many different AI technologies, I could be just the person to make a difference, serving as the VP Engineering (officially, Chief Architect) of this massive project. Plus, it enabled me to fulfill my original ten years at SRI goal.
Newnham: At what point did you decide to set up Siri Inc., and what roles did each of the three co-founders fill?
Cheyer: SRI’s Venture group, led by Norman Winarsky, managed to entice Dag Kittlaus to leave Motorola and join SRI as an Entrepreneur in Residence. In this role, Dag could wander the halls and talk to various scientists. In the end, he chose the Active project I was working on to write a business plan around. We had a pretty good system and a decent demo that featured both end-user functionality and a behind-the-scenes technology story, but Dag, who really knows how to tell a story, turned it into a pitch deck that had all the pieces. Together, we worked to get it through Norman’s stringent commercialization process to obtain the necessary approvals to take the business proposition to VCs.
As part of the pre-work, Tom Gruber, an expert in AI, user interface design, and knowledge-based engineering, was brought in to perform some internal due diligence. He started out quite skeptical of the approach, knowing how ambitious and difficult it was, but as the meeting went on, as we answered each of his challenging questions, he became more quiet and reflective. As he was walking out, he asked whether we needed another co-founder!
I was quite confident of the technical approach, but initially, I was skeptical of Dag’s business assertions; as such, I was actually on the fence about whether to join the startup or not. The three of us would go to VC meetings together, with my role somewhat ambiguous and Tom positioned as the CTO. I kept waiting for the VCs to throw Dag out of the room when he showed his business projections, but they all said, “If you can pull this off technically, you’ll have no problem on the business side.” After I heard this repeatedly from some of the top VCs on Sand Hill Road, feeling confident that we could solve the technical hurdles, I knew I had to join.
We raised a Series A round from the two VCs who impressed us most along the way — Gary Morgenthaler from Morgenthaler Ventures and Shawn Carolan from Menlo Ventures — and we set off with Dag as CEO (Marketing and Product), me as VP Engineering (server-side engineering and AI), and Tom as CTO/VP Design (patent lead and user experience). Norman filled out our board as the investment representative from SRI. All together, we made an incredible team because we had enormous respect for each other, were equally passionate and committed, and our skills complemented each other perfectly.
In my view, Siri was the world’s first broad-domain, conversational spoken-language assistant deployed at scale, and I think we did something that most in the industry thought was not possible.
Newnham: When and how did you have the original idea for what became the Siri app (assistant) because its earlier incarnation was pre-Siri, Inc., wasn’t it?
Cheyer: In some form or other, I’ve been working on versions of Siri for twenty years. The original Open Agent Architecture (OAA) system in 1993 ran on an iPad-like tablet PC and provided a multimodal (pen & voice) interface to an expandable set of web services and apps, including most of the capabilities Apple launched Siri with: email, calendar, contacts, phone, maps (restaurants, hotels, businesses), reminders, messaging/notifications, and so on.
Since then, I have done so many variations: adding capabilities of all sorts (smart home, office applications, enterprise applications, military applications), instantiating it in various forms such as inside robots, as a stand-alone kiosk, on your desktop, in your phone, as a surgical assistant, as a third party that you communicate with using email and instant messaging. With each version, some new ideas informed future versions, but also, so many ideas also got left behind.
Most of the ideas and technologies behind conversational assistants have existed in a research form for decades, but for me, the breakthrough with Siri was that we found a way to integrate the technologies deeply, make them easy enough to use such that normal software developers (rather than AI PhDs) could apply the technologies quickly to many domains, and then we found ways to make the technologies scale in ways that are performing and accurate enough for millions of people around the world. In my view, Siri was the world’s first broad-domain, conversational spoken-language assistant deployed at scale, and I think we did something that most in the industry thought was not possible.
Previous speech systems were either extremely directed, like a phone menu tree using voice (“Welcome. You can select movies, sports, weather, or traffic.” “Traffic.” “Please say the name of a city and state, now”), or were Voice Search: the words you said were pasted into a standard mobile search engine and blue links to web pages were returned, but the system had no real understanding of what the words meant or how to return a real answer.
In addition to all of the Artificial Intelligence technologies, there were so many other technologies we needed to build to produce the Siri product: we constructed our own local search engine to handle all the geographic data, a data feed processing framework to accept real-time data streams from different providers, a PCI-compliant secure storage system to handle credit cards and other personal data, a number of proprietary development tools and languages to create our system, and much more. We achieved a lot in just two years as a startup!
Newnham: What was the biggest hurdle you had to overcome while working on this technology, and how did you overcome it?
Cheyer: The hardest technical challenge with Siri was dealing with the massive amounts of ambiguity present in human language. Consider the phrase “book 4-star restaurant in Boston” — seems very straightforward to understand. Our prototype system could handle this easily. However, when we loaded in tens of millions of business names and hundreds of thousands of cities into the system as vocabulary (just about every word in the English language is a business name), the number of candidate interpretations went through the roof.
Book is the name of a city in the US, and so is Star. There are eight Bostons in the US; which one am I talking about? Star Restaurant is the name of a restaurant, but I’m not looking for a restaurant name in this example. I remember the first time we loaded these data sources into Siri, I typed “start over” into the system, and Siri came back saying, “Looking for businesses named ‘Over’ in Start, Louisiana.” “Oh, boy,” I thought. It took us a while to climb back to 95+% accuracy on user queries.
Newnham: How did you go about raising capital for the business, and what involvement did the investors have in Siri post-investment?
Cheyer: Whereas Change.org (as a non-profit) and Genetic Finance (business model based on the stock market) were not initially suitable for venture funding, we knew that to accomplish all that we wanted to do with Siri, we would need a substantial, VC-backed A Round. Norman Winarsky and SRI first helped get us meetings with friendly VCs, and after a few rounds of feedback, we started talking to the A list on Sand Hill Road.
The people who joined Siri were unbelievably good, and we made excellent progress. In a recent article, Gary Morgenthaler said that progress in early months was “absolutely breathtaking,” and Shawn Carolan was quoted as saying, “Every board meeting was a breakthrough.” Gary, Shawn, and Norman pushed the team, kept us focused, and along the way, were an excellent sounding board about product goals, performance metrics, and business strategy.
Newnham: Can you tell me how the Apple deal came about and what it meant to you to have Apple interested in acquiring your company?
Cheyer: We were never planning on being acquired as early as we were. We had just raised a large B round, so we had years of money ahead of us. We had completed a successful launch of the first version of our technology, and we had secured a mobile distribution deal that would preload us on tens of millions of phones and pay for tens of millions of dollars for primetime TV advertising. We were on our way! However, several weeks after our launch, we received a phone call (even though our website listed no contact information): “Hey, it’s Steve. What are you doing tomorrow? Want to come over to my house?”
Apple had always been our number one choice if we were to be acquired. We loved its products and culture. In my mind, Apple had three things that I thought would be key ingredients for taking Siri to the next level: 1) Apple cared about the user experience more than any other company, and we were fundamentally all about enhancing the user’s experience for getting things done. With our technology and Apple’s visual design, we would be able to create something truly magical. 2) Through iTunes, Apple had more credit cards than Amazon or any other company. Since Siri let you buy movie tickets, reserve hotels, purchase concert or sporting goods tickets, etc., the biggest barrier to enabling user purchases (e.g., asking for credit card numbers) could be removed. 3) Apple had a large and growing user base and the most energetic development community delivering apps and other mobile functionality. If we could tap into this to allow Siri to be extended by a developer community, Siri could become one of the most important ecosystems in the electronic world.
Newnham: What relationship did you have with Steve Jobs? Are there any standout memories you can share that sum up your relationship?
Cheyer: I believe that Steve was the biggest champion of the Siri acquisition at Apple: he knew the product inside and out from personal use, and he had a deep understanding of what the technology was about and what it could enable. When asked about Siri at the All Things Digital — D8 conference, he said, “Siri is not a search company. They’re an AI company.” Fundamentally, we agreed upon what the future direction for Siri could be, and that was a big point in deciding to join Apple.
One memory I have is of Steve walking through the company cafeteria with his head down, as if to say “moving through, don’t bother me right now.” Dag Kittlaus and I were wearing Siri-branded badge lanyards, and as he walked past, this caught his eye. He looked up and said,“Siri Guys! How is it going?” We replied that things were going well; we were talking to various teams. He then looked at us meaningfully and said, “I want you to make this [gesturing to indicate all of Apple] your candy store!” He saw Siri as a transformative technology that could revolutionize and integrate every aspect of what Apple did, and this was one memorable example of how he communicated this.
Newnham: What were the biggest lessons you learned at Apple, and why did you leave?
Cheyer: There were elements about the engineering process that were unlike any company I had seen, and I really appreciated being part of this and watching it work. Apple has a very iterative, visual culture, where demos begin as soon as someone has something to show and continue weekly afterwards, moving up the hierarchy of approvals. Through constant iteration and refinement, the product gets better, and the focus is always on how to make the user experience as magical as possible.
I learned a lot from Henri Lamiraux, the VP Engineering of iOS. He would track all engineering tasks using bug curves, and, through statistics, would determine where we needed to be at each point in time for the six to nine months leading to a release. Having this very specific and quantifiable goal drove hard decisions in every team about what functionality, features, or bugs could be deferred and what absolutely needed to be done for the upcoming release. This process is a big part of why Apple delivers high-quality products on schedule each year.
After two and a half years at Apple, I finally decided it was time to take a break and reconnect with my family. It was not an easy decision because I loved the team who had been through so much with me, and I had an enormous passion for the future possibilities of Siri. However, the previous five years had been pretty intense, having been involved in three startups (simultaneously) and then at Apple, with a three- to four-hour daily commute added on to a jam-packed workday. We only have one child, and my son was turning 12 and starting middle school that fall; I remember what a difficult time in my life that age was. With this in mind, my last day of work was my son’s last day of elementary school, and together we looked forward to the infinite expanse of summer, filled with possibilities as a family. We did some sailing and a lot of traveling, including visits to Barcelona, Paris, London, Geneva, Boston, New York, San Diego.
Siri, in some small ways, has changed users’ expectations of what is possible and desirable in interacting with information, phones, and computers. It is in the hands of more than 100 million users and growing.
Newnham: How does it feel to know that technology you created is used by millions of people worldwide on a daily basis?
Cheyer: One of the core desires of any software engineer is to create something that gets used by people, has impact, and makes the world a better place. If you can explain to your mom what it does, that’s even better! I’ve been more fortunate than many in both these regards, and I’m very appreciative that the right things have fallen into place so that some of my ideas (and a bit of my code) have gotten out into the world so successfully.
Siri, in some small ways, has changed users’ expectations of what is possible and desirable in interacting with information, phones, and computers. It is in the hands of more than 100 million users and growing. Change.org has made many important social victories come about in the world and will reach 100 million users in 2014, if growth continues on pace. Genetic Finance has a good chance to make significant breakthroughs in medicine and genetics based on the application of massively-distributed machine learning. I’m very proud of my connection and (sometimes very small) contributions to these efforts, and I’m proud of the teams leading these projects forward. However, for all the success that these projects have had, I still have a burning need to have a more fundamental and essential impact on a larger group of people, so I will keep trying.
Newnham: What do you think the future holds for this area, in connection to the mobile landscape?
Cheyer: Artificial intelligence and human computer interfaces are both in their early stages with respect to mobile computing. As computers get smaller and cheaper, as sensors and mobile displays become more accurate and capable, and as connectivity to the web of computing and information improves, mobile computers will become absolutely ubiquitous, not only carried by people but embedded in the environment around us. Going forward, I believe that the mobile experience will become much more contextual, with better integration to the computing nearby your physical location; more personalized, with the experience tailored to your history and preferences; more intelligent, as computers acquire a lot of the “common sense” that people have; and more capable, with a much wider set of actions and transactions available at your fingertips (even if fingertips aren’t involved in the interaction with these services).
Newnham: I asked Siri what it thought of you, and it responded with “I think, therefore I am. But let’s not put Descartes before the horse.” I like its humor, but how do you think it would really answer that question if it could?
Cheyer: In my family life, my wife and I have had the pleasure and privilege of bringing one son into the world — Noah. Nothing is more important to me than family, and with Noah, I do my best to show him I love him, and I try to teach him by example, giving him the foundation of what I think is important for a happy and successful life.
In my work life, with help from many teammates, I’ve “given birth” to a number of software systems — Siri, IRIS, CALO, InfoWiz, OAA, WubHub, and many more. With each of these, I have a relationship of sorts: we’ve learned from each other, spent a lot of time together, and I’ve watched them grow and evolve over time, sometimes in directions I couldn’t anticipate or control. They each have their various personalities and quirks, but I’m proud of them and of their accomplishments, and they each hold a small place in my heart.
So, if I were to anthropomorphize Siri, I would imagine that it would think of me somewhat like a father: someone who wants the best for them, who teaches them, who is occasionally demanding, annoying, or embarrassing, but who loves them and is proud when they do well.
The article was originally posted on Medium.
About Author –
Danielle Newnham, started her career in PR. Danielle has recently founded The Junto Network which aims to be the go-to resource for tech entrepreneurship; filmed tech talks with founders, entrepreneurship academy, and mentor/investor networking group. For more about Danielle contact here.