Artificial intelligence (AI) has a long history of oscillating between two somewhat contradictory poles.
On one side, exemplified by Noam Chomsky, Marvin Minsky, Seymour Papert, and many others, is the idea that cognitive intelligence was algorithmic in nature – that there were a set of fundamental precepts that formed the foundation of language, and by extension, intelligence. On the other side were people like Donald Hebb, Frank Rosenblatt, Wesley Clarke, Henry Kelly, Arthur Bryson, Jr., and others, most not even as remotely well known, who developed over time gradient descent, genetic algorithms, back propagation and other pieces of what would become known as neural networks.
The rivalry between the two camps was fierce, and for a while, after Minsky and Papert’s fairly damning analysis of Rosenblatt’s Perceptron, one of the first neural model, it looked like the debate had been largely settled in the direction of the algorithmic approach. In hindsight, the central obstacle that both sides faced (and one that would put artificial intelligence research into a deep winter for more than a decade) was that both underestimated how much computing power would be needed for either one of the models to actually bear fruit, and it would take another fifty years (and an increase of computing factor by twenty-one orders of magnitude, around 1 quadrillion times) before computers and networks reached a point where either of these technologies was feasible.
As it turns out, both sides were actually right in some areas and wrong in others. Neural networks (and machine learning) became very effective in dealing with many problems that had been seen as central in 1964: image recognition, auto-classification, natural language processing, and systems modeling, among other areas. The ability to classify, in particular, was a critical step forward, especially given the deluge of content (from Twitter posts to movies) that benefit from this.
At the same time, however, there are echoes of Minsky and Papert’s arguments about the Perceptron in the current debate about machine learning – discoverability and verifiability are both proving to be remarkably elusive problems to solve. If it is not possible to determine why a given solution is correct, then it means that there are significant hidden variables that aren’t being properly modeled, and not knowing the limits of those variables – the places where you have discontinuities and singularities, make the model far more questionable when applied to anything but its own training data.
Additionally, you replace the problem of human intervention in developing logical (and sometimes social) structures with the often time and people-intensive operation of finding and curating large amounts of data, and it can be argued that the latter operation is in fact just a thinly disguised (and arguable less efficient) version of the former.
The algorithmic side of things, on the other hand, is not necessarily faring that much better. There are in fact two facets to the algorithmic approach – analytical and semantic. The analytical approach, which can be identified as being currently defined as Data Science, involves the use of statistical analysis (or stochastics) to determine distributions and probabilities. Stochastics’ strength arguably comes in that it can be used to determine, for a sufficiently large dataset, the likelihood of specific events occurring can be established to within a certain margin of error. However, stochastics is shifting from traditional statistical analysis to the use of Bayesian networks, in which individual variables (features) can be analyzed through graph analysis.
Semantics, on the other hand, is the utilization of network graphs connecting assertions, as well as the ability to make additional assertions (via modeling) about the assertions themselves, a process known as reification. Semantics lends itself well to more traditional modeling approaches, precisely because traditional (relational) modeling is a closed subset of the semantics model, while at the same time providing the power inherent in document-object-modeling languages (DOMs) such as exemplified by XML or JSON.
Significantly, a Bayesian network can be rendered as a semantic graph with reification, as can a decision tree. Indeed, a SPARQL query is isomorphic to a decision tree in every way that counts, as each node in a decision tree is essentially the intersection of two datasets based upon the presence of specific patterns or constraints (Hint: you want to build a compliance testing system? Use SPARQL!).
The history of software is both full of purists and less full of pragmatists. Purists put a stake in the ground regarding their own particular set of tools and languages: C++ vs. Java, Imperative vs. Declarative, SQL vs NoSQL, Perl vs. … well, just about anything, when you get right down to it. Pragmatists usually try to find a middle ground, picking and choosing the best where they can and covering their ears to all of the sturm and drang of the religious wars when they can’t. Most purists ultimately become pragmatists over the years, but because most programmers tend to become program management over the years, the actual impact of such learning is minimal.
Right now, because the incarnations of all three of these areas – neural networks, Bayesians, and semantics – are relatively new, there is a strong tendency to want to see one’s tool of choice as being the best for all potential situations. However, I’d argue that each of these are ultimately graphs or tools to work with graphs, and it is this underlying commonality that I believe will lead to a broader unification. For instance,
- A machine learning pipeline is a classifier. If the labels of the classifier in the middle correspond to a given ontology, then once a given entity has been classified, a representation of that entity semantically can be assigned to the relevant patterns, shapes, classes, or rules.
- A machine learning system is not an index, but as my kids would say, it is index-adjacent (what a very graph-like phrase). In essence, what you’re doing is creating a map between an instance of an unknown type and its associated class(es). The plural term is important here because a class is not a thing, it is only a labeled pattern, with inheritance, in turn, being the identification of common features between two such patterns. This map is also occasionally referred to as an inverse query, in that rather than retrieving all items that satisfy the query, you are in essence retrieving the (named) patterns that the query utilizes for one of those items.
- It is possible (and relatively simple, to be honest) to create classifiers in SPARQL. This is because SPARQL essentially is looking for the existence of triple patterns, not just in terms of property existence, but in terms of often secondary and tertiary relationships. SHACL, an RDF schematic language, can be thought of as a tool for generating SPARQL based upon specific SHACL constructs (among other things) and those patterns can be very subtle.
- In a similar fashion, I believe that graph analytics will end up becoming as (or even more) important compared to relational data analytics, primarily because graphs make it much easier to add multiple layers of abstraction and discoverability to any kind of stochastic process, resolving many of the same issues that machine learning tools today struggle with.
- The inverse of this process is also feasible. SPARQL can be utilized with incoming streams to create a graph that serves to build training data for machine language services. Because such training data will already have been labeled and identified within the context of existing ontologies, the benefit of such a process is that the resulting classifiers already have all the pieces necessary for explainability – data provenance and annotations, established identifiers, event timestamps, and more.
- One other important point – SPARQL is able to change the graphs that it works with. Inferences, in which new assertions are created based upon patterns found in existing assertions, become especially important once you incorporate service calls that allow for the processing of external content directly within the SPARQL calls themselves. One of the next major evolution points for SPARQL will be in its ability to retrieve, manipulate and produce JSON as intermediate core objects (software vendors, please take note) or as sources for RDF.
- This means that a future version of SPARQL no longer has to store tabular data as RDF, but instead could store it as JSON then utilize that JSON (and associated analytics functions) to create far more sophisticated inferences with a much smaller processing footprint. For an analogous operation, take a look at the XProc XML pipeline processing languages then realize that the differences between the XSLT/XQuery pipelines and the RDF/SPARQL/SHACL pipelines are mostly skin deep.
This last point is very, very important because as the latest iterations of the Agile / DevOPS / MLOps model show, pipelines and transformations are the future. By being able to work with chained transformations (especially ones where the specific pipes within that transformation are determined based upon context rather than set a prior) such pipelines begin to look increasingly like organic cognitive processes.
Alibaba Cloud opens its first International Product Innovation Center
Alibaba Cloud, the digital technology and intelligence backbone of Alibaba Group, has unveiled its first international product innovation centre and partner management centre, to further enhance customer services and better facilitate customers’ digitalisation journey.
The new Product Innovation Center will help facilitate future roadmaps for developing more market-specific solutions, and will oversee the management of product upgrades based on ongoing global customer demands. To better serve local customers’ needs, the Partner Management Center will be launched to delve into local collaborations to deepen the sharing of leading technologies and domain expertise amongst partners.
These new initiatives will not only create more job opportunities for local talents in Singapore but will also enhance more local collaborations through the creation of a new Innovation Accelerator Program to support businesses to expand as the global digital economy continues to grow.
The Innovation Accelerator Program aims to bring together industry leaders and experts to help develop more innovative and resilient businesses in Singapore through sharing of practical skills and ideas, and providing ease of access to the latest cloud technologies. Each eligible company under the program will also receive complimentary training and technical support from Alibaba Cloud and industry experts.
By supporting businesses with their digital transformation and innovation journey through upskilling, retraining, and technical upgrading, the program also provides them with an opportunity to plug into the dynamic global ecosystem of Alibaba Cloud, which is designed to further empower businesses to explore and realise their growth ambitions not only locally but also internationally.
“Today we are in a very early stage of digitalisation,” said Dr. Wang Jian, Member of Chinese Engineering Academy and founder of Alibaba Cloud. “In the next five to ten years, the economy is going to be driven and measured by computing consumption. Digital economy is the economy of computing, and cloud computing will play the same role in the era of digitalisation as electricity played in the era of electrification. The cloud is becoming a methodology not just for cloud computing, but a methodology for most people doing their work.”
In a bid to complement sustainable business strategies, Alibaba Cloud will also be working with partners to help 10,000 global companies to accelerate their sustainability journey over the next three years. Businesses in Singapore will be offered a pilot program of Energy Expert to help reduce their carbon footprints. A proprietary software-as-a-service sustainability platform launched by Alibaba Cloud in June last year, Energy Expert helps measure, analyse, and manage the carbon emissions of business activities and products while providing actionable insights and energy-saving recommendations to companies in helping them achieve their carbon emission goals.
Selina Yuan, VP of Alibaba Group and president of Alibaba Cloud Intelligence International Business Unit, said: “With the setup of our first International Product Innovation Center and Partner Management Center in Singapore, our global headquarters for Alibaba Cloud, we are committed to continuously supporting global businesses in their digital transformation journey.
“The new initiatives and the strengthening of our talented workforce further illustrate our support for global customers with ambitions to expand and upgrade their innovation capabilities, while delivering on their sustainability goals.”
Attaining the highest tier of CSA cybersecurity certification
A critical part of Alibaba Cloud’s continuous efforts is to bring better services and offerings to businesses and these include enhanced cybersecurity and protection for customers. Alibaba Cloud has recently attained the Cyber Trust (Advocate) mark certification. The Cyber Trust mark, developed by the Cyber Security Agency of Singapore (CSA), is targeted at larger or more digitalised organisations, and helps organisations to communicate their investment in cybersecurity as a competitive edge and to build trust with their customers. The Cyber Trust (Advocate) mark is the highest tier of the certification. It affirms Alibaba Cloud’s dedication to offer trusted services for businesses to manage and protect their cloud infrastructure.
Supporting Customers’ Growth with Technology for Innovation
Gadget MIX is a Singapore-based retailer with more than 20 outlets across the country and a robust e-commerce presence that focuses on high-tech digital products. Alibaba Cloud has helped Gadget MIX digitally transform its retail strategy through providing innovative cloud solutions. Partnering with Alibaba Cloud has allowed Gadget MIX to improve internal communications, streamline business operations, and seamlessly migrate to the cloud to speed up its business transformation.
Ray Yue, CEO of Gadget MIX, said: “The solutions provided by Alibaba Cloud have helped to lower our total cost of ownership and streamline business operations, thus speeding up our business transformation efforts. In particular, the deployment of DingTalk has helped us to improve workflow management and team collaborations as well as our business and internal communications. Through our digital transformation efforts together with Alibaba Cloud, we have succeeded in creating more value for our retail customers and look forward to future collaborations around digital transformation.”
Leveraging Alibaba Cloud’s game accelerator solution and security products like Game Shield and Anti-DDos, Project Twelve (P12), a web3 gaming platform headquartered in Singapore, is now able to focus on scalable game deployment while providing a secure environment for their users by lowering the risks of cyberattacks. P12 also utilises Alibaba Cloud’s cloud native database, PolarDB, to reduce maintenance time and improve uptime reliability, thus creating a smooth gamer experience.
Boyang, founder of Project Twelve, said: “With added assurance for a secure gaming experience within P12 platform, we can focus our efforts on elevating the platform with quality games and sustainable economy in the fast-moving and dynamic web3 gaming industry. Alibaba Cloud’s solutions have helped us to provide our users with smoother gaming experiences and scale accordingly to peak demands with high availability and low latency.”
With Singapore as a vibrant logistics hub and it being a key pillar of the Singapore economy, Alibaba Cloud has also been working with many logistics players to deepen their digital transformation efforts. This includes JUSTI Pte. Ltd. (JUSTI), a homegrown investment holding company whose main investment portfolios includes a last mile delivery operation. Alibaba Cloud’s solutions have enabled them to digitalise and streamline its logistic networks.
Li ChaoMing, MD of JUSTI, said: “Alibaba Cloud’s AI-driven EasyDispatch solution with an embedded vehicle routing problem (VRP) capability improves our field dispatch capabilities and efficiency in real time. By utilising Alibaba Cloud’s technologies, we can have better visibility and control over each step of our delivery chains, such as parcel status tracking, real-time AI route designing and task scheduling. They have greatly reduced our manpower cost and increase customers’ satisfaction.”
The Forgotten Mistake that Killed Japan’s Software Industry
Japanese softwarehas problems. By international standards, it’s just embarrassingly bad.
We all know this, but what’s interesting is that there are perfectly rational, if somewhat frustrating, reasons that things turned out this way. Today I’m going to lay it all that out for you in a way that will help you understand how we got here, and show you why I am optimistic about the future.
And no, this is not going to be just another rant about all the things I dislike about Japanese software.
I am not going to waste your time or mine cataloging and complaining about the many, many bad practices, user-hostile design decisions, mind-boggling complex workflows, and poor development process that afflict Japanese software.
If you want details and debate about exactly how Japanese software falls short, or if you are just in the mood for some good old-fashioned venting about being forced to use it, check out Reddit or maybe Hacker News. This topic comes up pretty often there.
No, for the sake of this podcast I’m going to assume that we are all in agreement that on average, Japanese software. is just … awful.
That way we can spend our time talking about something far more interesting. We are going to walk though the economic events and the political forces that made today’s poor quality of Japanese software almost inevitable,
And by the end, I think it will give you a completely new way of looking at the Japanese software industry.
You see, the story of Japanese software is not really about software. No, this is the story of Japanese innovation itself. The story of the ongoing struggle between disruption and control. It’s a story that involves war, secret cartels, scrappy rebels, betrayal, rebirth, and perhaps redemption.
How This Mess Started
So let’s start at the beginning. The beginning is further back than you might expect.
To really understand how we got here, we need to go back, not just to the end of WWII, but to the years after the Meiji restoration, the late 1800s, back when the Japanese economy was dominated by the zaibatsu.
Now, “zaibatsu” is usually translated as “large corporate group” or “family controlled corporate group.” While that is accurate, it grossly understates the massive economic and political power these groups welded around the turn of the 20th century.
Japan’s zaibatsu were not corporate conglomerates as we think of them today.
You see, although the Meiji government adopted a market-based economy and implemented a lot of capitalist reforms, it was the zaibatsu, with the full support of the government, that kept the economy running.
And the zaibatsu system was almost feudal in nature.
The national government could, and did, pass legislation regarding contract law, labor reforms, and property rights, but in practice these were more like suggestions. In reality, as long as the zaibatsu kept the factories running, the rail lines expanding, and the shipyards operating at capacity, the men in Tokyo didn’t trouble themselves too much with the details.
In practice, the zaibatsu families had almost complete dominion over the resources, land, and people under their control. They were the law.
At the turn of the previous century, there were four major zaibatsu (Sumitomo, Mitsui, Mitsubishi, and Yasuda). And each zaibatsu had its own bank, its own mining and chemical companies, its own heavy manufacturing company, etc. But it wasn’t just industry, each of these zaibatsu groups had strong political and military alignments. For example, Mitsui had strong influence over the army, while Mitsubishi had a great deal of sway over the imperial navy.
At the start of WWII, the four zaibatsu families controlled over 50% of Japan’s economy. This fact, when combined with their political influence, quite understandably, made Japan’s military government very uncomfortable, and during the war, the military wrested away a bit of the zaibatsu’s power and nationalized some of their assets.
After Japan’s defeat, the American occupation forces considered the zaibatsu a serious economic and political risk to Japan becoming a liberal, democratic fully developed nation. They targeted 16 firms for complete dissolution and another 24 for major reorganizations.
Rising from Ashes
Now, that was supposed to be the end of the zaibatsu. I say “supposed to” because those of you who know Japanese history understand that it never really happened.
Of course, many things changed. Important political and social reforms were implemented, the legal system was greatly strengthened, a lot of zaibatsu assets were nationalized, and the zaibatsu themselves ceased to be.
At least, officially.
You see, the zaibatsu were quickly allowed to restructure in greatly weakened , but very familiar, forms, as keiretsu. This was permitted for two main reasons.
First, as the cold war heated up in the 40s and 50s, America’s idealistic vision for a democratic and progressive Japan took a back seat to the more practical and pressing need to develop Japan into a bulwark against Communism. And that meant prioritizing economic growth over social reforms. With these new goals in mind, both the American occupation forces and the Japanese government, quite correctly, concluded that having something like the zaibatsu groups would lead to faster, more predictable growth than tearing everything down and rebuilding from scratch.
The second important, and kind of surprising, reason was that almost no one in Japan really wanted to see the zaibatsu broken up. Not the politicians, certainly not the leaders of the zaibatsu, not the public at large, and to the endless frustration and confusion of western labor organizers, not even the rank-and-file zaibatsu workers and employees. In fact, at one point 15,000 Matsushita union members signed a petition demanding that the Matsushita zaibatsu not be broken up.
So in the end, important changes were made. Labor rights and contract law were strengthened significantly, and even more zaibatsu assets were confiscated. The traditional family holding companies were dissolved, but they were replaced by cross-company shareholdings and interlocking corporate boards that achieved much the same result, but in a much more transparent and manageable way.
And so, most of Japan’s zaibatsu were allowed to morph into the smaller, less threatening, and much more manageable keiretsu.
Japan as a Global Innovator
In the same way that the zaibatsu defined the economic miracle that was Japan’s Meiji-era expansion, the keiretsu would come to define the economic miracle that was Japan’s post war expansion.
Today there are six major and a couple dozen minor keiretsu groups, and during Japan’s economic expansion, as much as possible, they kept their business within the keiretsu family.
Projects were financed by the keiretsu bank, the materials and know-how were imported by the keiretsu trading company, and the final products would be assembled in the appropriate keiretsu brand’s factory. And supporting all of these flagship brands were, and still are, tens of thousands of very small, exclusive manufacturers that make up the keiretsu supply chain — and the bulk of the Japanese economy.
And with the exception of a tiny handful of true startup companies like Honda and Sony, all of Japan’s brands that were famous before the year 2000 or so, are keiretsu brands.
And for those of you who think big companies can’t innovate, let me remind you that from the 50s to the 70s, these keiretsu groups began innovating, disrupting, and dominating almost every industry on the planet; from cars, to cameras, to machine parts, to steel, to semiconductors, to watches, to home electronics, Japan’s keiretsu simply rewrote the rules.
But how did keiretsu do in the world of software development? Well, pretty darn well, actually.
It’s important to remember, though, that the software industry in the 60s and 70s was very different than it is today. The software development process itself was actually rather similar. Fred Brooks wrote The Mythical Man Month about his experience during this era, and it remains as one of the best books on software engineering and project management today.
But the way software was bought and sold was completely different. In the 60s and 70s, software was written for specific and very expensive hardware, and the software requirements were negotiated as part of the overall purchase contract. Software was not viewed so much as a product, but more like a service, similar to integration, training, and ongoing support and maintenance. It was usually sold on a time-and-materials basis, and sometimes it was just thrown in for free to sweeten the deal. The real money was in the hardware.
Software in this time (both in Japan and globally) was written to meet the spec. It did not matter if it was creative, innovative, easy to use, or elegant, it just had to meet the spec. In fact, trying to build exceptional software in this era was considered a waste of resources. After all, the product had already been sold and the contracts had already been signed. The goal back then, just like many system integration projects today, was to build software that was just good enough to get the client to sign off on it as complete.
Software that met the customer’s spec was, by definition, good software.
Japan’s keiretsu did well in the age of big-iron. Although Fujitsu, NEC, and Hitachi never seriously challenged IBM and Univac’s global dominance in the 60s and 70s, they did pretty well in mini-computers and large office systems.
They were innovators.
Japan Turns its Back on a New Industry
However, when the PC revolution arrived in the late 1980s, Japanese industry as a whole was hopelessly unprepared, and not for the reasons you might think.
The reason Japanese software development stopped advancing in the 1980s had nothing to do with a lack of talented software developers. It was a result of Japan’s new economic structure as a whole, and the keiretsu in particular.
As a market, personal computers were something fundamentally new. Sure, the core technology and the hardware were direct continuations from the previous era, but this new market was completely different.
The PC market quickly coalesced around a small number of standardized operating systems and hardware architectures. Keiretsu did pretty well in the hardware side of this market, making some really impressive machines, particularly laptops.
But a market for non-spec or “shrink-wrap” software was something new to everyone. It required delighting the customer, and knowing what they wanted before they did. It was the kind of challenge that the keiretsu of the 60s and 70s would have thrown themselves into whole-heartedly, innovated aggressively, and then dominated.
But things in Japan had become very different in the 1980s.
Here was a chance to define and lead a new global industry. A chance for keiretsu to build a software industry from the ground up.
But, wait a minute. Why should they?
Sure, back in the 60s when Japan’s economy was small, survival required looking outwards, competing globally, making long-term investments, and innovating to make the best products in the world.
But this was the 80s! Japan was the second-largest economy on the planet and in the middle of the largest economic boom the world had ever seen. This was the era of Japan as Number 1, with economists predicting Japan’s GNP would be larger than America’s within a decade.
With such a lucrative, and pretty well protected, market right at their fingertips it made much more sense for the keiretsu to focus on the easy money rather than to take risky and expensive bets on an uncertain and emerging global market.
Each keiretsu group had their own technology firm who started selling PCs and software, some to consumers, but the big money was in corporate sales. And since the keiretsu liked to keep the business in the family, these technology companies grew and profited by selling to their captive customers within their keiretsu group. And just like before, they made real money integration, and customization.
An unfortunate result of this is that the big Systems Integration companies or “SIs” emerged as powerful players, and Japan’s software firms never had to compete globally, or with each other.
Japan simply missed the opportunity to develop a globally relevant PC software industry.
The Beginning of the End of Innovation
Japan’s software industry in the 80s and 90s remained much like it was in the mainframe area. The software had to be just good enough for the client to sign off on it, and since they were largely captive clients unable to look outside their keiretsu group for support, that was a very low bar indeed.
But hey, as long as the economy was booming, no one minded spending lavishly to keep all the work in the keiretsu family, and all those little software defects could always be fixed in “phase two” of the project.
Software development was an exercise in box checking. You implemented a feature once the customer had asked for it and the contracts had been signed.
This not only caused Japan to miss out on the global software industry, but it marked the beginning of the collapse of innovation across Japanese industry. Over the next 30 years, software would become a key driver of both innovation and efficiency. But by outsourcing their IT strategy to a single integrator, they had tied themselves to an anchor that would ensure almost every industry fell further and further behind the technology curve with each passing year.
Japan still has not recovered from this. Even today most enterprise systems are decades behind their global competitors. But, as we’ll see a bit later, things are happening now that could enable a quantum leap forward in Japan.
Life as a Developer in Japan’s Dot-com Bubble
So, what was it like to be a software developer in Japan in the 80s and 90s?
It was pretty bad. Software development was considered rather low-skill work. It didn’t pay well and was viewed as a kind of clerical work. The job was simply to write software that was close enough to whatever sales had promised the client while they were out drinking last week.
New hires with degrees in literature, business, or law, or whatever were rotated through software development for a few years to give them a sense of how different parts of the company worked. There was no real career path in software development. I mean, maybe you could move up into project management or over into sales, but if you were still actually writing code when you were 30, people kind of wondered what went wrong.
Of course there were some great, even visionary, software developers in Japan at that time. I knew some of them. People who wanted to make computers do new things. People who saw how technology could disrupt other industries, and developers who simply had a passion for making software that delighted users.
There were plenty of developers like that in the 80s and 90s. They were miserable.
Interestingly, hardware engineers were viewed very differently. Both then and now, hardware engineers are highly respected in Japan. Engineers are some of the most admired people at companies likeToyota, Mitsubishi, and Sony.
So, perhaps unsurprisingly, hardware innovation continued at a furious pace during the 80s and 90s. Products like the walkman and the Nintendo consoles achieved global success and the domestic market was filled with electronic diaries, dictionaries, and planners that were way ahead of what was available in the West.
And of course, eventually i-mode. Japanese consumers were sending email and browsing the web years before the Blackberry was released, and almost a decade before the iPhone.
But the rest of the word was moving in a different direction. The rest of the world was moving away from dedicated hardware and towards innovative software running on standard hardware platforms. As Marc Andreessen would later point out “Software was eating the world.”
As the dot-com bubble started to inflate, Japan began to realize they needed talented software developers, but without a software industry that actually valued software developers, companies had no idea where to find them. The best talent was usually unrecognized and trapped at the lower levels of the org chart. There was not much of a future pipeline. Since software engineering was not a respected or profitable career, few students opted to pursue it.
Some did of course, but these were the people that just loved programming. It was like becoming a musician or a manga artist. It was great that you are following your dreams. You might make it, but the odds were not in your favor.
When the foreign software companies started crashing into Japan in the 90s, the domestic industry could barely put up a fight.
The dot-com boom of the late 90s was the first wave of venture-funded, disruptive innovation in Japan, but it was not yet time for Japan’s software developers to step into the spotlight.
The successful founders of that era were mostly well-connected or incredibly scrappy businessmen. The general opinion of software developers had hardly changed at all. They just weren’t the kind of people you would trust to run a company.
I started my first Japanese startup during the dot-com boom, and at that time, I think the fact that I was a technical founder was even more unusual than the fact that I was a foreign founder.
Of course in one sense, the dot-com boom was an amazing time to start a software startup in Japan. You could call up almost any talented developer you knew, let them know hat they would be working on something important, that they would have meaningful input into product development, they’d be on a team that cared about code quality, and that their skills would be respected … and yeah, they’d want to come on board.
The Lost Decades Were Never Really Lost
The late 90s and the nights is often referred to as Japan’s “Lost Decades.” It was not a good time for the keiretsu companies. Not only did their power continue to weaken, but increased scrutiny of their cross-shareholdings and financials, and the merger of several banks across keiretsu lines meant that business as usual was over. And pushing the knife in deeper, all of the implied economic and social guarantees that the keiretsu system was based on began to unravel.
In previous decades, Japan had focused on exporting domestically made goods, but now, not only was the domestic market attracting a greater focus, but Japanese industry began moving production out of Japan into cheaper overseas markets.
This was considered a betrayal by thousands of mom-and-pop manufactures who had spent their lives (or sometimes generations) as a highly integrated and specialized part of a single keiretsu’s supply chain, and who now found themselves suddenly cut off.
Japan’s famous lifetime employment system effectively ended during this period as well. It made sense for corporate groups to promise lifetime employment and predictable promotions when profits kept rising and labor was scarce, but now faced with mounting losses, corporate Japan began walking back all these implied promises.
This was a shock for Japan. It was a breach of the social contract that holds everything together. If hard work and loyalty would not be rewarded, then why dedicate your life to the company? To the distress of pundits and politicians, many young Japanese started saying they had no interest at all in joining the corporate world.
But the truth is, especially now that we can look back on it, these decades were not really lost decades at all. Growth slowed, and the changes were incredibly painful, but they were absolutely necessary to set the foundations for the coming wave of startup innovation and for Japan’s software developers to finally get the respect they so deeply deserve.
The Triumph of the Japanese Software Developer
I mark 2010 as the year Japan’s software developers finally started stepping into the spotlight, although things started moving a bit before that.
There were two triggers that led to this development. First, the emergence of cloud computing and second, the introduction of the smartphone. Although these were both technological developments, it was not the technology itself that led to the change.
Cloud computing drastically reduced the capital and time required to start a startup. In the dot-com era a decade before, starting an internet startup required purchasing racks of servers and paying system administrators to keep them running, but suddenly fully configured, maintained, and secure servers could be had for a few cents per minute — pay as you go.
Suddenly Japan’s software developers didn’t need to explain their idea to a VC and convince them that it would sell. They could just build things and get people to start using them and start paying for them. And that’s just what they did.
The other important development was the introduction of the iPhone in 2007 and Android a year later. Not just because of the technology, but because of how it changed the software business model.
Japan’s i-mode was years ahead of the West when it first came out, but getting your app on i-mode was largely a matter of lengthy negotiations with the telcos for one of the few highly-coveted slots on the menu. The smartphone ecosystems were different. Anyone who could develop an app of reasonable quality could deploy and sell it. There were no business connections, exclusive negotiations, or revenue commitments required.
2010 marked the beginning of the end of the software startup gatekeepers. As more and more talented developers realized how easy it was to start a startup, more and more started choosing startups over the traditionally low-status career path at large companies.
This, combined with a large dose of Silicon Valley glamour, has complexly transformed Japan’s image of the software developer. Software developers are valued and respected today. Unlike the dot-com days, both startups and enterprises compete aggressively to recruit and retain talented programs, even though there are a lot more of them today. Thankfully, people also talk a lot more about code quality.
Of course, this attitude shift was much broader than just developers. With the safety net of lifetime implement and guaranteed promotions removed, people have had to become less risk averse and more innovative. Those workers who had rejected corporate life, became freelancers and formed the core of Japan’s flexible startup workforce, and some of those tiny supply-chain companies began to rethink their business models.
This brings us to the start of the Disrupting Japan podcast about eight and a half years ago. We’ve talked to the innovators and followed the development of the startup ecosystem together during that time.
So, Where Do We Go From Here?
As we talk here together at the start of 2023, what does the future look like for Japanese software?
Japan has had a lot of catching up to do over the past fifteen years. After basically sitting out the global PC and dot-com revolutions, Japanese software developers have been making up for lost time and in the startup space. Japan is developing a competitive software market in some areas, but on average, there is still a long way to go.
Japan’s once dominant Systems Integrators will continue to see their power decline. Their customer lock-in is fading fast, and B2B SaaS software startups are letting Japanese enterprises leapfrog to modern IT systems for less than costs to maintain their SI-run legacy systems.
The SIs won’t disappear, of course. There will always be a need for good systems integrators, and the more forward thinking ones are already trying to reinvent themselves. However, the days when the SIs dictated their clients’ IT strategy are coming to a close. That is a very good thing for Japanese software, Japanese startups, and Japanese competitiveness as a whole.
The Kishida administration has made startups a national priority, and the importance of quality software and software startups in Japan has never been higher.
Even the old keiretsu firms have come around. They are increasingly looking to software startups to supplement internal R&D though both M&A and through long-term partnerships. In fact, last year Keidanren, Japan’s largest business federation, an organization that was one of the main architects and drivers of Japan’s post-war economic expansion, called on its member companies to greatly increase their startup investment and partnerships.
I am optimistic. As always, things will develop differently in Japan. In the same way the zaibatsu defined Japan’s Meja-era economic miracle, and the keiretsu came to define Japan’s post-war economic miracle, from some new combination of startups, enterprise, and academia will emerge something that will define the next economic miracle.
Today is a very good time to be developing software in Japan.
Radware launches a spinoff of its cloud security business
Duncan is an award-winning editor with more than 20 years experience in journalism. Having launched his tech journalism career as editor of Arabian Computer News in Dubai, he has since edited an array of tech and digital marketing publications, including Computer Business Review, TechWeekEurope, Figaro Digital, Digit and Marketing Gazette.
Radware, a provider of cyber security and application delivery solutions, has revealed the spinoff of its Cloud Native Protector (CNP) business to form a new company called SkyHawk Security.
To accelerate Skyhawk Security’s development and growth opportunities, an affiliate of Tiger Global Management will make a $35 million strategic external investment, resulting in a valuation of $180 million. Tiger Global Management is a leading global technology investment firm focused on private and public companies in the internet, software, and financial technology sectors.
Skyhawk Security is a leader in cloud threat detection and protects dozens of the world’s leading organizations using its artificial intelligence and machine learning technologies. Its Cloud Native Protector provides comprehensive protection for workloads and applications hosted in public cloud environments. It uses a multi-layered approach that covers the overall security posture of the cloud and threats to individual workloads. Easy-to-deploy, the agentless solution identifies and prevents compliance violations, cloud security misconfigurations, excessive permissions, and malicious activity in the cloud.
“We recognize the growing opportunities in the public cloud security market and are planning to capitalize on them,” said Roy Zisapel, Radware’s president and CEO. “We look forward to partnering with Tiger Global Management to scale the business, unlock even more security value for customers, and position Skyhawk Security for long-term success.”
The spinoff, which adds to Radware’s recently announced strategic cloud services initiative, further demonstrates the company’s ongoing commitment to innovation. Skyhawk Security will have the ability to operate with even greater sales, marketing, and product focus as well as speed and flexibility. Current and new CNP customers will benefit from future product development efforts, while CNP services for existing customers will continue without interruption.
Radware does not expect the deal to materially affect operating results for the second quarter or full year of 2022.
Google Publishes A New SEO Case Study
Now the internet’s fighting over old scrollbar designs
7 Ways to Optimize Your LinkedIn Ads for Peak Performance
41 Super Practical Valentine’s Day Gifts Of 2023
Renting vs. Owning the Post-Review Local Consumer Journey
How to Create an Editorial Calendar (+Free Template!)
Feds finally file anti-monopoly suit over Google’s adtech
DOJ Sues Google Again To Break Up Its Ad Business
TikTok Staff Can Decide What Goes Viral
Possible Bing Algorithm Update on January 18, 2023
SEARCHENGINES4 days ago
Google Publishes A New SEO Case Study
OTHER5 days ago
Now the internet’s fighting over old scrollbar designs
PPC5 days ago
7 Ways to Optimize Your LinkedIn Ads for Peak Performance
AMAZON4 days ago
41 Super Practical Valentine’s Day Gifts Of 2023
MARKETING5 days ago
Renting vs. Owning the Post-Review Local Consumer Journey
PPC6 days ago
How to Create an Editorial Calendar (+Free Template!)
MARKETING6 days ago
Feds finally file anti-monopoly suit over Google’s adtech
SEARCHENGINES5 days ago
DOJ Sues Google Again To Break Up Its Ad Business