5 Error

Building a Multi-Tenant Dev Environment

Tue, 14 Jul 2026 00:00:00 GMT

Every engineering org that grows eventually hits the same bottleneck: testing changes in a shared dev environment becomes unreliable very quickly. Everyone deploying to the same environment means everyone stepping on each other’s data. When the frequency of deployments ticks up, the problem gets worse. With AI, people are able to create features faster, and it will continue to get worse. After each feature, you typically want to test a flow end to end, but someone else’s test just mutated the state you depended on while you were testing. Such an environment earns a reputation for being unstable, and nobody trusts it.

This will typically spark demand for isolated environments, and it has happened where I work in multiple ways, on different forums. The ask is always the same. How do I spin up a full copy of the system, mutate it however I want, throw it away, and get a fresh one, without affecting anyone else?

Some context on what “the system” means here. We run a microservice architecture on Kubernetes, with multiple teams owning several services, each with a single delegated task. We also depend on a Postgres database, Kafka, a secrets vault and most things you would find on a modern cloud-native platform. Let’s call the whole running system a world. No service in it works alone: any critical flow cuts across many of them. One of many critical flows is straight-through order processing : everything that happens between a customer clicking buy and the platform confirming the stock is theirs. A single order touches a good number of services end to end. If that path breaks, the business feels it immediately. And when an engineer says they want to test their change properly, this is usually the flow they mean, which makes it exactly the thing a shared environment is worst at.

Testing that flow in isolation is worth a lot, today. The first reason is smoke tests. A test that runs the chain end to end for every pull request, from any of the services involved, massively increases confidence in the change you are about to unleash. Without isolation, that confidence comes slowly: you deploy through shared stages and watch your work crawl through the pipeline before you learn anything. An isolated world collapses that loop to seconds, and that is a powerful thing a platform engineer can hand to product engineers. The second reason is load testing. You want to bootstrap a world, seed it with a particular scenario, and run a load test that targets a specific use case. If you do not seed a version of the world, your results rely on whatever data happens to be in place, and that comes with problems of its own. Runs stop being comparable, because the data shifted underneath them, and the accumulated junk looks nothing like production, so the numbers describe the leftovers from previous runs, rather than the system.

But the reasons go beyond what we can do today. The most valuable version of this tool is not one that runs the system as it is, but one that lets you inject your own changes, your branch, your local image, into a running world and test against everything else. That is a major hassle today, and it is exactly what a disposable world makes cheap. And one reason only became obvious recently: agents. Long-running agentic workflows are blocked by the same shared environments humans are. An agent that wants to validate its own changes needs a human in the loop before it can touch anything shared, and that human is the bottleneck. Give the agent a disposable world and the verification loop closes without anyone babysitting it. Agents need sandboxes even more than humans do.

So I set myself a deliberately idealistic goal. I should be able to run multiple parallel worlds, each involving many microservices, side by side, ideally on my laptop, sharing infrastructure but with fully isolated data planes. Data generated in world A must never leak into world B.

This post is about the four main problems that stood between me and that goal, and how I solved them.

Why now

Among other internal factors, one thing changed fundamentally: the cost of building tools like this has collapsed, and AI is the reason. It also helps that I have a pretty handy AI budget that makes experiments like this possible.

Think about the traditional economics of a project like this. An isolated environment tool touches Kubernetes, databases, secrets infrastructure, and Kafka internals. Under the old model that is a multi-quarter effort. Design documents, reviews, prototypes, more reviews, and months of iteration before anyone can touch a working version. Projects like this rarely die because they are bad ideas. They die because the upfront cost never clears the bar against everything else competing for engineering capacity. That is why demand can sit unanswered for years.

AI changed the shape of that cost curve. I sat down with an AI assistant, had it help reason through each implementation one after the other, and iterated. Try an approach to TLS termination, watch it fail, understand why, try the next one, all within an afternoon instead of a sprint. I never left the model to run amok or reduced it to result delivery. I was constantly the human in the loop, stubbornly intentional and deliberately slow about design choices. The decisions remained mine while the drafts got cheap. The expensive part of building software has always been the iteration loop: the time between having an idea and learning whether it survives contact with reality. AI compresses that loop brutally. A few weeks of this, and I had a working system spanning four domains that would each have been a project on their own.

An interesting consequence is not that a tool got built faster. It is that a whole class of projects that used to be economically unjustifiable is now viable. If there is an internal tool your team has wanted for years, the math may have changed. It is worth rerunning the numbers.

But I digress.

The core

The naive approach is to duplicate everything per world, just like we have in our current setup. Every database, every broker, every secret store, cloned for each environment. That is slow, ridiculously expensive, and does not scale.

So I flipped it on its head. Share all the infrastructure, and make every layer multi-tenant. One Postgres. One Kafka. One secrets endpoint. Each world gets a partitioned slice. This is to be a fresh, self-contained stack, stood up once per machine. The infrastructure can be as simple as docker-compose; the tool’s job is to provide the rails for bootstrapping isolated worlds on top.

There is one constraint I fought hard to keep: services must not know any of this is happening. No code changes. No tenant awareness. No forcing teams to adopt a particular client library. If a service needs modification to run in a world, the design has failed. Adoption dies the moment you put homework on someone else’s desk.

To that end, there were four main problems I had to solve: manifests, databases, secrets, and Kafka.

Manifests

Production Kubernetes manifests are built for high availability. Autoscalers , PodDisruptionBudgets , service mesh sidecars, observability agents, generous resource requests. None of that matters for a throwaway dev world, and most of it actively fights you in a local dev environment.

So I strip manifests to the bare bones. Autoscalers and disruption budgets are removed. Replica counts are forced to one. Resource requests are set to cluster defaults, so nobody gets to ask for twelve CPUs. The mesh sidecar goes. The observability agent goes, for now, though I suspect putting logs and metrics back in a dev-friendly form will be a real quality of life win later.

I set this up as a transformation pipeline. Source manifests go in one end, an ordered series of transformations runs over them, and stripped-down dev manifests come out the other. That decision paid for itself. A pipeline is testable and tested, each transformation is composable, and when a later problem needs something injected into every manifest, the injection is just one more stage. It shows up again twice below.

What is left is a deployment, a service, and the external secrets the service depends on. Deploy that into a local kind cluster and it works out of the box.

Postgres

In production, each service effectively gets its own managed Postgres instance, with backups, failover, and authenticated access. For local dev you need none of it, and duplicating an instance per service per world is a non-starter.

The nice thing is that Postgres is already multi-tenant by design, and it gives you two units of tenancy to choose from. The obvious first stop is schemas : one database, one schema per world, table names stay stable, and search_path does the routing. It was the first thing I tried, but that did not work. Some services ship migrations that point explicitly at the public schema, and that assumption runs deep. Fixing it means changing services, and changing services is against the rules.

So I went one level up. The unit of tenancy became the database name, which no service ever references in code … yet. It arrives through configuration, which makes it the one knob I can turn without anyone noticing. I run one Postgres instance, and every world gets its own set of databases, prefixed with the world name. World A gets world-a.service-x. World B gets world-b.service-x. Both live side by side.

Services already take their connection settings from environment variables: host, user, password, database. Injecting the prefixed database name per world is just another stage in the manifest pipeline from above, and requires zero code changes. Two worlds on one instance, and neither is any the wiser.

Secrets

This was one of the harder problems. But in the end, the solution was incredibly obvious. Services fetch secrets from the cloud provider’s Secret Manager, either through the SDK or through Kubernetes External Secrets . Both paths hit the same public endpoint, over TLS, authenticated with cloud identity.

You cannot point that at localhost. Or can you?

The trick was a three-layer fake. First, DNS. Inside the local cluster, we rewrite DNS so the Secret Manager hostname resolves to a secrets emulator instead of the real cloud API.

Second, TLS. The SDK expects a valid TLS handshake, and the emulator does not speak TLS. So the setup creates its own root certificate authority, appends it to the trust store of every pod it ships, as part of the pipeline, and uses it to issue a certificate for the Secret Manager hostname. Every pod now trusts the fake endpoint, because we made it so.

Third, gRPC. Secret fetching happens over gRPC, so after terminating TLS you forward the gRPC traffic to the emulator.

All of this is nginx configuration.

The service calls the same hostname it calls in production, gets a valid handshake, speaks gRPC, presents the same credentials it always presents, and receives its secrets. It never learns that everything behind the hostname is fake. As a bonus, that internal CA became a reusable building block. Once you can issue certificates that every pod trusts, you can impersonate any TLS endpoint you need to mock. It shows up again in the next section.

Kafka

Kafka is fundamentally not multi-tenant. There is one global topic namespace per cluster. Create topic-a and everyone on the cluster sees topic-a.

The official Kafka guidance is to prefix all topics with a tenant identifier and enforce it with policies. Fine, except our services are not tenant aware, and by my own constraint they are not allowed to become tenant aware. So something has to sit between the service and Kafka and do the tenancy transparently.

I had four requirements for that something. A single shared broker, with no tearing infrastructure down and up per tenant. Support for the actual Kafka wire protocol, not just produce and consume but metadata, offset commits, and transactional IDs. Light enough to run alongside everything else. And licensing that does not require a procurement conversation for an experiment.

I evaluated what the market had. Kroxylicious was the most viable option I found. It is open source and works as designed, but it runs on the JVM and kept getting heavy next to everything else. Conduktor Gateway is easier to use, but the features I actually needed sit behind an enterprise license, and an experiment is not the time for that conversation. The third option, making services tenant aware, is technically the correct answer and my least favorite. It forces an opinion on every team, requires everyone to adopt a specific client library, and means waiting for org-wide adoption before the tool works at all.

Having worked on some large-scale Kafka deployments in the past, and with some experience in encoding formats , I rolled up my sleeves and, with a touch of madness, wrote my own. I called it kroxy. It started as a weekend project, and is internal for now.

kroxy does one thing. It sits between the service and Kafka, figures out which tenant a connection belongs to, rewrites topic names with the tenant prefix on the way in, strips the prefix on the way out, and proxies everything else untouched. Transactional IDs get the same treatment, because Kafka transactions carry identifiers of their own. No ACLs, no permissions, no event hooks, none of the enterprise features the market proxies offer. Transparent tenancy rewriting at the protocol level, and nothing else.

When you start a world, a provisioner registers the tenant with kroxy and creates the prefixed topics on the real broker. Connection settings are injected as part of the manifest transformation pipeline, in the shape each service already expects. A service asks kroxy for some-topic. kroxy translates that to world-a.some-topic, talks to the shared broker, and translates back on the response. The service believes it is talking to a normal Kafka with its normal topics.

This is also code that cannot be quietly wrong. kroxy sits on the wire protocol, exercised by every client on every request. Implementing it carefully went a long way toward making bugs easy to track down and fix. Of which there were a few.

It worked.

Running it

I started on my laptop, and for a handful of services it was fine. But a full world in our setup is over a hundred services, and by the end it was clear the complete stack was too heavy for the machine I type on. You can run it. You will not enjoy it.

So I ported the implementation to a VM in Google Cloud with Packer and Terraform , and designed the split deliberately: your laptop remains the authoring surface, the VM becomes the backend. You write, the VM runs. The binary syncs over, and the whole stack lives there under systemd: kind for service container orchestration, plus containerized Kafka, Postgres, the secrets emulator, and kroxy, managed with nerdctl as a replacement for docker compose.

The end state is four commands. Two to build and start your VM, one to initialize the stack, one to spin up a world. After the first run, spinning up a new world is a single command.

This is v1. I foresee it evolving to a unified setup where a big cluster hosts all worlds and dedicated providers back all shared infrastructure. But for now, this works.

What I did not build

Scope discipline was half the project.

The obvious end game is PR preview environments. Open a pull request, get a world, see the URL in the PR. Everything above paves the road there, but that feature drags in CI, compute capacity, and provisioning questions that have not been answered yet. The aim was to test the idea by figuring out the building blocks first.

Data seeding is the other gap. An empty environment is not that useful. You need baseline data before you can exercise anything real. It is a real problem, it is on the roadmap, and I have intentionally not solved it yet.

And worlds are throwaway by design. So updating one is as simple as throwing what you currently have away and conjuring a new one, which always reflects the current state of the system. Throwaways also keep the blast radius honest: the worst any bug can do is break a world you were going to delete anyway. Migrating a lingering world forward is a convenience feature for later, if ever.

The takeaway

If you take away nothing else from this, remember: multi-tenancy beats duplication, and transparency is the design constraint that decides adoption. Every solution above, the database prefixes, the fake Secret Manager, kroxy, exists so that services stay completely unaware they are living in a partitioned world. And AI is making experiments like this cheaper to run.

The next time someone tells you an environment cannot be isolated without cloning everything, ask which layer is actually not multi-tenant. The answer is usually shorter than the excuse.

We are always looking for interesting people to solve interesting problems with. If that is you, Upvest is hiring .

What is in a String... In Go

Fri, 23 Jan 2026 00:00:00 GMT

Strings are ubiquitous. You use them everywhere; Logs, APIs, user input, configuration, serialization. They feel like a primitive type. You declare them, concatenate them, pass them around, and it all just works.

C doesnt have strings. In Java, strings are not a primitive type. That is because strings are not as simple as int or bool. Under the hood, a string is a composite structure: a pointer to some bytes and a length. Go does a lot of heavy lifting to make strings feel seamless, and that is a good thing. But this convenience has a cost. Most engineers never take the time to understand how strings actually work, and they end up writing code that is slower than it needs to be.

For a type that feels so simple, that is surprising. But once you understand what is happening under the hood, it makes perfect sense.

That is what we are going to try to unpack.

Before we get there, we need to talk about two things: garbage collection and slices. These two concepts are critical to understanding why strings are what they are in Go, and why the choices you make when manipulating them matter. It also bears to note that, while the concepts presented here are specific to Go, they are broadly applicable to most programming languages even though the implementation details might be different.

Go has two memory regions where your data can live: the stack and the heap.

The stack is the easy one. When you call a function, Go allocates space for local variables on the stack. When the function returns, that space is automatically reclaimed. No cleanup required. It is fast and cheap.

The heap is different. When data needs to outlive the function that created it, or when the compiler cannot determine the size at compile time, that data goes on the heap. The heap is managed by Go’s garbage collector. The garbage collector periodically scans the heap, figures out what is still being used, and frees what is not.

This is where things get interesting. Modern Go uses a concurrent, tri-color mark-and-sweep garbage collector. While it does have brief stop-the-world phases, most of its work runs concurrently with your program. However, the more garbage you create, the more work the collector has to do, and the more CPU time it consumes.

The key insight here is that allocations are not free. Every time you allocate something on the heap, you are creating future work for the garbage collector. This might seem like a minor concern for small programs, but at scale, GC pressure becomes one of the primary performance bottlenecks. It is impossible to prescribe where your variable should go. That is the job of escape analysis, but depending on how you write your code, you can “make a convincing argument” to the escape analyzer at compile time.

Arrays in Go are fixed-size, value types. ie, the size is a part of the type. When you pass an array to a function, the entire thing gets copied. Because of this, arrays are rarely used directly. Instead, we use slices.

A slice is a descriptor that points to an underlying array. Under the hood, a slice is a struct with three fields (see reflect.SliceHeader):

type SliceHeader struct {
    Data uintptr  // pointer to the underlying array
    Len  int      // number of elements in the slice
    Cap  int      // capacity of the underlying array
}

When you create a slice like this:

s := make([]byte, 3, 5)
s[0], s[1], s[2] = 'a', 'b', 'c'

You get a slice header pointing to a backing array of 5 elements, with the first 3 populated. The slice knows its length is 3 and its capacity is 5.

This design has a consequence that trips people up: multiple slices can share the same backing array.

arr := [5]byte{'a', 'b', 'c', 'd', 'e'}
s1 := arr[0:3]  // [a, b, c]
s2 := arr[1:4]  // [b, c, d]

Both s1 and s2 point to the same underlying array. If you modify s1[1], you are also modifying s2[0]. They overlap.

The other thing to understand is what happens when you append beyond capacity. If you have a slice at capacity and you append to it, Go allocates a new, larger backing array, copies all the elements over, and returns a new slice pointing to this new array. The old array becomes garbage.

s := make([]byte, 3, 3)  // len=3, cap=3
s = append(s, 'd')       // new backing array allocated

This is important. Every time you exceed capacity, you pay for an allocation and a copy. The old backing array is now garbage waiting to be collected.

Now we can talk about strings.

A string in Go is a read-only slice of bytes. Under the hood, it looks like this (see reflect.StringHeader):

type StringHeader struct {
    Data uintptr  // pointer to the underlying bytes
    Len  int      // number of bytes
}

Notice what is missing: there is no Cap field. A slice has pointer, length, and capacity. A string only has pointer and length.

Why no capacity? Because strings are immutable. You can never append to a string in place. You can never modify its contents. If you cannot grow a string, you do not need to track how much room there is to grow into.

This is what Go hides from you. When you write s := "hello", you are not just storing five characters. You are creating a struct with a pointer to those bytes and a length. The simplicity is a facade.

This immutability is a deliberate design choice. It makes strings safe to share across goroutines without locks. It makes them safe to use as map keys (see the Go spec on string types). But it comes with costs.

What happens when you concatenate two strings?

s := "hello"
s = s + " world"

Since strings are immutable, Go cannot modify the original string. Instead, it allocates a new backing array large enough to hold both strings, copies the bytes from both into the new array, and returns a new string pointing to it. The old backing array becomes garbage.

Every concatenation is an allocation and a copy.

This seems fine for a single concatenation. But consider this:

var s string
for i := 0; i < 1000; i++ {
    s += "a"
}

How many allocations happen here? One thousand. Each iteration allocates a new backing array slightly larger than the last, copies all the existing bytes plus the new one, and discards the old array.

But it gets worse. This is not just O(n) allocations, it is O(n^2) copying. On the first iteration, you copy 1 byte. On the second, 2 bytes. On the third, 3 bytes. By the end, you have copied 1 + 2 + 3 + … + 1000 = 500,500 bytes. For a 1000-byte string.

This is why string concatenation in a loop is one of the most common performance mistakes in Go. And it is why strings.Builder exists.

var b strings.Builder
for i := 0; i < 1000; i++ {
    b.WriteString("a")
}
s := b.String()

strings.Builder maintains an internal byte slice that grows using the same doubling strategy as append. When you call String(), it converts the byte slice to a string without copying, using an unsafe trick. One allocation for the final string, plus a handful of allocations as the internal buffer grows. Orders of magnitude better.

If you know the final size ahead of time, you can do even better:

var b strings.Builder
b.Grow(1000)  // pre-allocate
for i := 0; i < 1000; i++ {
    b.WriteString("a")
}
s := b.String()

Now you have exactly one allocation.

Go does some clever things to reduce allocations for string literals. When you write:

a := "hello"
b := "hello"

Both a and b point to the same backing array. The compiler interns string literals. It recognizes that these are the same string and stores it once in the binary’s read-only data section. No heap allocation at all.

But this only works for compile-time constants. If you build the same string at runtime:

a := "hel" + "lo"  // compiler folds this to "hello", interned
b := strings.Join([]string{"hel", "lo"}, "")  // runtime, not interned

The first one gets interned because the compiler can evaluate the concatenation at compile time. The second one happens at runtime, so it allocates.

This also has implications for string comparison. Comparing two strings is O(n), Go compares byte by byte. If you are using strings as map keys, every lookup pays this cost. For short strings, this is negligible. For long strings used as keys in hot paths, it adds up.

Converting between strings and byte slices is another source of hidden allocations.

s := "hello"
b := []byte(s)  // allocates

Why does this allocate? Because strings are immutable and byte slices are not. If b shared the same backing array as s, you could do b[0] = 'X' and suddenly s would contain “Xello”. That would break the immutability guarantee.

So Go copies. Every time you convert a string to a byte slice, you get a new allocation. Every time you convert a byte slice to a string, you get a new allocation.

b := []byte("hello")  // allocation
s := string(b)        // allocation

There are unsafe ways around this if you really need them, but you should not reach for them unless you have profiled and identified this as an actual bottleneck. The compiler also optimizes some cases. For example, []byte(s) used immediately in a read-only context may avoid the copy. But do not count on it.

If you take away nothing else from this article, remember this: every string modification is a heap allocation, and heap allocations are GC work.

When building strings incrementally, use strings.Builder. Pre-size with Grow() if you know the final length.

var b strings.Builder
b.Grow(estimatedSize)
for _, part := range parts {
    b.WriteString(part)
}
result := b.String()

If you need the result as []byte rather than string, use bytes.Buffer instead:

var b bytes.Buffer
b.Grow(estimatedSize)
for _, part := range parts {
    b.WriteString(part)
}
result := b.Bytes()

If you are holding a small substring of a much larger string, be aware that the substring shares the backing array of the original. The large string cannot be garbage collected until all substrings are gone. Use strings.Clone() to get an independent copy:

large := getLargeString()  // 10MB string
small := large[0:10]       // still references the 10MB backing array
small = strings.Clone(small)  // now has its own 10-byte backing array

For high-throughput scenarios where you are creating many short-lived strings, consider sync.Pool to reuse buffers:

var builderPool = sync.Pool{
    New: func() interface{} {
        return new(strings.Builder)
    },
}

func process(parts []string) string {
    b := builderPool.Get().(*strings.Builder)
    b.Reset()
    defer builderPool.Put(b)
    
    for _, part := range parts {
        b.WriteString(part)
    }
    return b.String()
}

Strings feel simple, but they are not. Under the hood, a string is a pointer and a length, and every modification means a new allocation. Go chose to make strings immutable, which makes them safe and easy to reason about, but it means every modification creates garbage.

The mental model is straightforward: string modification equals heap allocation equals GC pressure. Once you internalize this, you start seeing the problems before they bite you.

The next time you see a + inside a loop, pause. That is probably where half your allocations are coming from.

The day the internet run out of addresses

Fri, 17 Oct 2025 00:00:00 GMT

In 1981, Jon Postel handed out /8 blocks like they would last forever. Take 16 million addresses, he said. We will never run out. MIT took one. So did Ford, IBM, and a few dozen others. By 2011, IANA gave away the last block. Postel had been dead for 13 years.

This is a story of how we got here, how the internet went from abundance to scarcity. From Postel’s paper notebook to IP address marketplaces. From end-to-end connectivity to NAT traversal hell. It is a story about optimism meeting reality, about choosing convenience over architecture, and about how technical debt compounds across decades.

The Age of Abundance (1981-1990)

When RFC 791 standardized IPv4 in September 1981, the internet was tiny. We are talking about connecting research institutions and military networks, not billions of smartphones. The 32-bit address space seemed infinite. 4.3 billion addresses for a network that had maybe a few hundred hosts.

Jon Postel, one of the internet’s founding architects, managed address allocation with what legend describes as a paper notebook. There were no contracts, no formal procedures, just an informal understanding that you would do the right thing for the good of the internet. If you needed a network number, you asked Jon. He would write it down and that was that.

The classful addressing system introduced in RFC 790 divided the address space into three classes. Class A networks got a /8, 16.7 million addresses each. Class B got /16 blocks with 65,536 addresses. Class C got /24 blocks with just 256 addresses. The assumption was straightforward: big institutions get big blocks, smaller ones get smaller blocks.

MIT got 18.0.0.0/8 in 1979, before classes even existed. Stanford got one. So did companies like IBM, Ford, and Apple. The US Department of Defense ended up with multiple /8 blocks. When you are building something new and the pool seems bottomless, why not be generous?

Here is the thing though. Even then, some people knew this would not scale forever. The decision to use 32 bits was not arbitrary. It was a tradeoff between header size and address space. But in 1981, the idea that we would have billions of connected devices seemed like science fiction. The practical concern was getting the damn thing to work at all.

The First Cracks (1990-1993)

By 1990, the cracks started showing. The IETF’s Network Working Group noted that with rapid escalation in the number of networks and concurrent internationalization, address allocation needed serious rethinking. Class A and B addresses were becoming an increasingly scarce commodity whose allocation must be handled with thoughtful care.

The problem was twofold. First, classful addressing was wasteful. If you needed 2,000 addresses, you would get a Class B with 65,536 addresses and waste 63,000 of them. Or you would get eight separate Class C blocks and fragment the routing table. Second, the routing table was growing faster than routers could handle.

In RFC 1338 published June 1992, the IETF proposed supernetting, what would become CIDR (Classless Inter-Domain Routing). Instead of fixed-size classes, you could allocate any power-of-two block size using variable-length subnet masks. A /20 here, a /22 there, sized to actual needs. It was elegant, it was flexible, and it bought us time.

But CIDR was a bandaid. It made allocation more efficient and helped with routing table growth, but it did not solve the fundamental problem. We were still burning through a finite address space, just more slowly.

The NAT Compromise (1994-1998)

This is where we made a choice that still haunts us. In May 1994, RFC 1631 introduced Network Address Translation. The idea was simple: use private address space internally (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) and translate to public addresses at the network edge.

NAT was supposed to be a temporary workaround. It became doctrine.

With NAT, suddenly you did not need public addresses for every device. One public IP could serve hundreds or thousands of internal hosts. ISPs loved it because they could serve more customers with fewer addresses. Network admins loved it because it simplified management. Security folks loved it because it added a layer of obscurity, though not real security, despite what people still claim.

But NAT broke the internet’s end-to-end principle. The original design assumed any host could directly communicate with any other host. NAT made that impossible. Want to run a server from your home connection? Good luck. Need peer-to-peer connectivity? Welcome to STUN, TURN, ICE, and a baroque collection of hole-punching techniques.

We spent the next 25 years building increasingly complex workarounds for a workaround. Every VoIP protocol, every P2P application, every WebRTC implementation, they all have to dance around NAT. We normalized brokenness and called it good enough.

Meanwhile, IPv6 was being developed. The specification came in 1998, the same year Google was founded. It offered 340 undecillion addresses, enough to give every atom on Earth’s surface a trillion IPs. It fixed NAT. It improved routing. It was everything we needed.

We did not deploy it.

The Exhaustion Era (2008-2011)

By the late 2000s, the writing was on the wall. IANA exhausted its primary address pool on February 3, 2011, allocating the last five /8 blocks to the regional internet registries. APNIC ran out in April 2011. RIPE followed in September 2012. ARIN implemented a waiting list in 2015.

We knew this was coming for years. The predictions varied on timing, but nobody thought the IPv4 space would last indefinitely. Yet here we were, with IPv6 specified for over a decade and adoption still minimal.

Why? Inertia, mostly. IPv6 is not backward compatible with IPv4. You cannot just flip a switch. Every router, every firewall, every application that hardcodes IPv4 assumptions, it all needs updating. That is expensive. That is risky. That requires coordination across millions of independent actors.

And crucially, the people who controlled IPv4 address space had zero incentive to push migration. Why would they? Scarcity made their assets valuable.

The Market Era (2011-Present)

When Nortel sold IPv4 addresses to Microsoft in 2011, something shifted. IP addresses became tradable commodities. Today, a single IPv4 address costs $40-60 on secondary markets. A /24 block goes for $10,000-15,000. That /8 MIT got for free in 1979? Worth nearly a billion dollars at market rates.

Stanford gave their /8 back to APNIC in 2000, apparently out of altruism. MIT sold parts of theirs to Amazon in 2017. The regional internet registries now explicitly allow transfers subject to some conditions. We have financialized scarcity.

And here is the uncomfortable truth: this probably slowed IPv6 adoption. When AWS can charge for elastic IPs, when holding IPv4 space is a balance sheet asset, when entire companies exist to broker address transfers, the status quo works just fine for some very powerful players. Every “we will support IPv6 when our customers demand it” statement is just protecting rent-seeking infrastructure.

Meanwhile, we keep building workarounds. Carrier-grade NAT, where ISPs put entire neighborhoods behind a single public IP. IPv4-to-IPv6 translation layers. Dual-stack deployments where everything runs twice. We have made the simple act of addressing a device on a network into a multilayered complexity nightmare.

Where We Are Now

IPv6 adoption sits around 40% globally as of 2025, which sounds decent until you realize we have had 27 years. Some countries are doing better. India is over 60%, driven by mobile carriers. The US hovers around 50%. But huge swaths of the internet remain IPv4-only.

The good news? We are past the tipping point. Major cloud providers run IPv6 internally and translate to IPv4 for legacy systems. Mobile networks deploy IPv6-only with translation for IPv4 destinations. New devices ship with IPv6 enabled by default. The migration is happening, just very slowly.

The bad news? We have wasted decades and billions of dollars on complexity we did not need. Every STUN server, every NAT traversal library, every dual-stack deployment, that is engineering effort that could have gone to solving actual problems instead of routing around artificial scarcity.

What We Can Learn

This is not really a story about IP addresses. It is a story about how we respond when we hit resource limits. We had three options when IPv4 exhaustion became inevitable:

Migrate to IPv6 aggressively
Make IPv4 more efficient (CIDR, NAT)
Create markets to allocate scarcity

We chose options 2 and 3. We optimized the existing system and financialized the constraint. We did not choose option 1 until scarcity was already baked in and powerful interests had formed around maintaining it.

The pattern repeats everywhere in tech. When faced with fundamental limits, we tend to build increasingly complex workarounds rather than addressing root causes. Technical debt is not just about code. It is about architecture, about systems, about the compounding cost of deferred decisions.

But here is what gives me hope: the internet has always been surprisingly resilient. We have made questionable decisions, we have optimized for the wrong things, we have let institutional inertia win battles it should not have won. And yet the thing keeps working. It keeps growing. It keeps evolving.

IPv6 is happening. Not as fast as it should have, not as cleanly as it could have, but it is happening. In another decade, maybe two, IPv4 exhaustion will be a historical footnote. Kids learning networking will wonder why we ever thought 4.3 billion addresses was not enough. The NAT workarounds will fade into legacy systems nobody wants to maintain.

The internet’s original sin was not using 32-bit addresses in 1981. That was a reasonable engineering tradeoff given the constraints. The sin was clinging to that decision for 30 years after we knew better. Not for technical reasons, we had the solution, but because changing course required coordination and someone not getting paid.

In the end, we are getting there. Just very, very slowly. And at tremendous unnecessary cost. But we are getting there.

Further reading:

Unfortunately, Redis

Tue, 18 Jul 2023 00:00:00 GMT

Redis is designed to be a blazingly fast key-value in-memory database that trades some durability for speed. And for most parts, it lives up to the billing. Setting up and using Redis has always been a breeze. The success and popularity of Redis has made it a very powerful tool in the software engineers’ arsenal; with its support of almost all data types and associated operations out of the box, it is a perfect solution for almost all persistence problems. As with everything, people have used it in ways without considering its fitness for the problem at hand, often to great success.

Redis is famed for being able to perform operations that is measured in sub milliseconds. And the key to this is how it is designed. It is designed to store data in memory. If your first-year computer science class is anything to go by, you will know this provides much faster read and write throughput compared to disk by miles. When we create data structures using variables, we often use arrays, hashes and maps. Redis supports these including esoteric ones like hyperlog, bloom filters and more. It is also efficient at manipulating them because it takes advantage of the underlying data structures that are optimized for in-memory storage without worrying about how to persist it to durable storage. It is also single-threaded, Though a performant single-thread system might sound counterintuitive, there are some peculiar performant advantages to it. And Redis takes advantage of this in a brilliant way to ensure consistency without any cost to performance. Redis’s single-thread will scale indefinitely in terms of I/O concurrency. It does this by using an I/O demultiplexing mechanism and a concise event loop designed by the author. Thus there is no synchronization to be done since all commands are serialized. It might look like the CPU might become a bottleneck with this design, but it turns out you will often hit a network bottleneck well before the CPU cannot keep up. The positive side effect of this design is that the atomicity of all operations comes at no extra cost. Redis also uses a proprietary protocol that is much more terse. Couple the isolated event loop with a proprietary communication protocol, and you have a blazing fast in-memory data store that scales indefinitely in theory, and you have a highly performant database.

These facts only hold when the size of your payload and the number of connections remain relatively small. This easily jumps out the window with ever-increasing load parameters. The threshold is, unfortunately, rather low at a high number of connections and increased payload sizes. Modern large-scale micro-services will easily have over 100 running instances at medium scale. And since most instances employ some pooling mechanism so as not to pay a connection cost for each request, a single Redis instance is going to do a bit of work in maintaining those connections, not to talk of serving requests as they come through. To improve performance at medium to high loads, some projects such as KeyDb, Snapchat’s drop-in replacement as an alternative to Redis, employ a multithreaded approach and a bit of magic to sustain some high workloads. This has been touted to provide 5x performance over Redis. Another solution which I have seen used is to employ a proxy that multiplexes over multiple redises. One such proxy is the twoemproxy developed at Twitter. Twoemproxy, or nutcracker as it is informally known, is in itself single-threaded and employs key hashing to store keys in shards of multiple redis instances, giving you proper multiplexing. While it may look like this is susceptible to the original problems of a single-threaded application, it is not necessarily the reality because twoemproxy employs a single thread for each Redis instance, turning the whole system into a multi-threaded system. Of course, this is still susceptible to hotkeys. These might look like ideal solutions, but setting up new infrastructure as an intermediary service introduces a new failure point, which is neither trivial nor ideal. But when done right, there are a lot of net positives.

Another major concern with Redis is durability. Redis out of the box does not persist data on disk, only in memory. This means when a server goes down, so does all your data. Durability is serious business and when it becomes a priority, this is where Redis starts to go backwards. Redis was never planned to provide durability beyond RAM. This is evident in the fact that disk persistence was never part of Redis until v0.04. Redis supports two types of persistence modes.

The first one is called snapshotting or RDB. When snapshotting is enabled, Redis will periodically write all your dataset in memory to disk. This is good for point-in-time recovery. But this also means you lose all the data between when the last snapshot was created versus when the failure occurred. For a moderately busy server, there are bound to be significant changes that happen between when snapshots are set, and losing it might not be a good thing. To combat this, some teams set the snapshot times as minimal as possible to combat the amount of data it is possible to lose. This can be a bad idea when your data set is considerably large. Writing a 1gig file to disk every 60 seconds is a recipe for disaster. What happens in the background is Redis forks a child process for background processing, serializing the dataset in memory, making it disk compatible, writing it to a temporary file in the background and rename the file atomically upon finish. Even though the overhead of creating a fork is zero in theory when the OS supports copy-on-write, you still need to turn on the overcommit_memory. This is because if the dataset between the parent process and the child process deviates, Redis will not keep track of the changes and will have to allocate just as much memory your data set has to the child process in order for the snapshot to be successful. In snapshotting mode, you must do everything to ensure that your datastore does not exceed half the RAM allocated to Redis, otherwise, your Redis server will implode with an OOM. This is the default persistence mode because it is simple and safe for small data sets. When your data set starts to increase considerably in size, think twice about snapshots.

The second more durable persistence mode is Append Only File, introduced in Redis 1.1 to solve the drawbacks of snapshotting. In this mode, every Redis command is appended to a file as a log. Very much like WAL logs for conventional RDBMSs. This way, you can build the entire database by replaying the entire file. It can be argued that sequential writes to files are significantly faster than random access but is is still significantly slower than writing in memory, and this goes against the essence of what Redis is, which is an in-memory data store. If you are going to be writing each command to a file, why not use a datastore that is designed for that in the first place? If you use AOF, it means Redis is going to call fsync at a point in time which can be configured in 1 of 3 ways. appendfsync always will call fsync on each command. This is very very safe. But with this option, you might as well throw Redis out the window because the performance becomes inferior to every database designed to fsync on write. If you use this mode without a Battery Backed Write Cache (BBWC) RAID controller, you will get fucked. Have fun figuring out what went wrong. appendfsync everysec will call fsync every second, which means, at most you lose a second of data. This might sound reasonable, but it is not without its drawbacks. If you have an update-intensive application like a counter that updates many times in a second, you end up with a needlessly huge AOF file when in essence, the data payload might only have a small footprint. That said, it is an easier choice and has been the new default since Redis 2.4. appendfsync no delegates the calling of fsync to the operating system. This is the fastest and the least safe method amongst all the appendfsync options. Normally Linux will flush data every 30 seconds with this configuration, but it’s up to the kernel’s exact tuning.

When I talk to other developers about systems design problems, most people are quick to suggest Redis as a solution to buffer data when producers can not keep up with consumers, leaning heavily on the high write-throughput of Redis. The solution often goes like this; use Redis as a cache layer between the two entities. While this design generally works, I often wonder if people truly understand the tradeoffs and risks involved here. The lack of proper durability in Redis makes the bandwidth of solutions it is a fit for a very narrow one; nonetheless, for that bandwidth, it offers tremendous advantages. If you will drop Redis into your stack, be sure it is absolutely what you need, or stay clear of it. Otherwise, you and your data are going to get burnt.

Sharing Crazy Bold Ideas

Tue, 21 Jun 2022 00:00:00 GMT

In 2007 I got into Mfantsipim High School and my tech journey began. My school was nicked named Botwe, after the hill on which it is situated. We further shortened it to B-Town. I started to pick up programming, I started with ruby and html. It was a boarding school and computers were not allowed. So we wrote out programs on scrap notes and typed them out on the computers when we got the chance. That was once a week.

Fast forward to 2 years later and I came up with this bold crazy idea to develop a video game for my high school. I called it B-Town Vice City. The concept was simple: create a game very much like the much famed Vice City at the time with the setting as my high school. I had less than 500 hours of programming experience. I knew nothing about video games or 3d modeling. Or anything for that matter. I told a bunch of guys in the computer club who also thought it was a bold idea but they liked it so we got to work trying to learn everything there was about making video games.

Word slowly went out that Bubu and the Computer Club were making a video game; B-Town Vice City. I became a laughing stock. Everywhere I went, people pointed fingers at me and laughed at me; That is the guy who wants to build Vice City. It became so bad that some guys in the club started to distance themselves from the effort because we had become the latest laughing stock. People would see me and ask me how my game was going basically to elicit laughter from their group of friends basically at my expense. It bothered me at first but nonetheless, I persisted. I slowly began to grow a thick skin. I don’t know where the conviction came from but I was set on developing a video game and nothing anyone would say would deter me. And for a long time, nothing did. Now that I look back I think I would probably have laughed at myself back then.

Those of us who were left on the effort decided to abandon the mission due to reasons that had nothing to do with the fact that we were laughed at or that the idea was too audacious. We went on to do different things. But for years the laughter followed. Even past university. Even today, some mates from high school see me today and casually remind me about how I dared to build a game 14 years ago. But these days it is more in nostalgia than it is in ridicule. So we laugh.

I have since gone on to share bold ideas where ever I find myself. A lot of people laugh but soon stop when they realize oh shit, he is serious. I won’t lie, the majority of my ideas are ridiculous, if not batshit crazy. But that doesn’t stop me. Because a while ago, I realized the power of sharing your ideas. Your crazy idea might not get considered, but nobody will ever forget that you were the one who dared share them in the first place. And this alone counts for everything. And if any of your crazy ideas ever do get taken seriously, it is almost always a thrill.

Share your crazy ideas wherever you find yourself. You might not know who is listening. People will never forget the idea or the person who dared to share.

The Difficult Truth about Mentorship

Tue, 11 May 2021 00:00:00 GMT

Many people who know me one way or the other knows one thing; I work tireless at getting new and upcoming devs to get mentored. This comes from a personal story.

When I began with computers in high school, the only resources I could consult were an old delphi manual, a unix bible, a general purpose programming book and 2 other friends(Sam and Victor). Most of the things I picked up within the next 2 years were all from these resources. Fast forward to university where I was studying computer engineering and I was so happy at the fact that I was in a group where the average person knew more than I did and was ready to learn from my peers. Or so I thought. It turns out, my peers all had exactly the same questions I had. So we hacked until we got answers. And sometimes for days. But we enjoyed the process. That alone was enough to keep us going.

When I got out of uni and saw the outside world, I was immediately enthralled by all these guys who had been doing the things I so much wanted to be doing. They also had what seemed like the right answers to my most burning sofware engineering related questions I had had for years. What blew me away was the realization at the fact that if I had met these “seniors” about a year or two earlier the trajectory of my path would have been completely different. Growth would look more like a hockey stick rather than a drying line. That is also when it dawned on me that, new developers need mentorship and it was the only way through which we could build capable software engineering capacity early enough to cater for the onslaught of sofware engineering roles coming to Africa. 5 years on and this realization has never been truer.

Most new developers I have spoken to know they need mentorship and are often eager to get under the wings of a senior even if their time spent is not paid for. Most of them are often surprised when they realize nobody is willing to take them on even though they are ready to render free service. There is a hard truth about mentoring new devs that nobody really tells you.

I will not talk about what the structure of a mentorship should look like, albeit informal or not. I fear that is a conversation beyond the scope of this essay.

The very thing new devs looking for mentorship fail to understand is that the cost involved in being under someones wing is not only how much you get paid. This is even the least of the problems. The cost involved has more to do with the senior you want to understudy. Seniors are expensive and companies would rather spend money to have then do company work rather than mentor a junior who is not directly employed by the company. This means even if a senior does decide to mentor you, it is going to be on his own time. More often during his downtime, and those hours I reckon are even more expensive. Time spent with a mentee is time not spent with his/her family or loved ones or doing things that he would rather be doing. There are people who are fortunate to find seniors ready and willing to put in the effort to mentor them. If you are ever fortunate to find one who is willing to put in all this effort for you, hold them in high regard, remain forever grateful and never dissapoint them. Remember you are one of the lucky few and don’t forget to pay it forward.

Where does that leave everybody else. The average joe out there will never find a dedicated mentor. That is not to say they should not be mentored. Infact, if you diassociate the notion of finding a mentor from that of being mentored you start to realize that mentorship can come in any shape or form. The onus is on you to be able to effectivly utilize the mentorship channel availed to you.

And this is where DevCongress comes in. DevCongress is a free, open to all, slack group that you can join by creating an invite for yourself. I strongly believe in DevCongresses ability to avail a channel through which individuals can get mentored. I have seen it play over countless times for numerous people including myself. Yes, DevCongress is not perfect, neither is the world. But I believe everybody who is part of DevCongress is working strongly to make it a better place for everyone. Some times discussion get heated and can get personal, I think that demonstrates the passion in there more than anythig else. And when it does Admins do step in to ask everyone to take a step back and cool off. If you are willing to look past these imperfections, identify where you need help, be bold enough to phrase your questions and ask them, you will never be dissapointed in the quality of the responses you get. Be it cyber security, (@finixbit has you covered), php (@silentworks has you covered), javascript, (@theRealBraZee has you covered) and everything inbetween, the community is welcoming, ready and willing to answer what ever questions you have.

I believe there are two main reasons DevCongress works as a mentorship channel. The first of which is the excellent community of seniors who remain dedicated to break down complex topics to anyone willing to listen. The second and most important reason is the decentralized way with which we provide help. Anybody at anytime can chip in and answer a question at their own time. It being a 5 min break when an app is compiling or as a way to wind down in the evening. There is never pressure to check work and people can pick their own hours. I have realized this makes is easy for people to help even if the help drags on for a few hours.

If you are a developer within the African ecosystem, i entreat you to join DevCongress and be part of the community. You will be easily astonished at how much it will give back to you. I know because I am still being blown away.

This article was prompted after having a chat with @umfrumf

Thanks to Andrew Smith for editing this piece.

A scalable development process - Part 1

Mon, 12 Aug 2019 00:00:00 GMT

Every so often when we build apps, one question we constantly try to answer is “How will this app scale?”. This leads us to take some steps and measures to optimize our apps in the hope that when we get 5,000 users on launch day or 5,000 api request per second on launch day, our servers dont go down. Unless you operate at the scale of Google or Facebook, you will not get 5,000 user on launch day. So if you build to handle traffic from 5,000 people or 5,000 requests per second it becomes premature optimization. And in the words of Donald Knuth himself, premature optimization? It is the root of all evil.

One thing programmers often overlook is how well their development process scales over time. I think rather than write your apps to scale from the get go, you should create a development process that scales. Throughout the life of an existing application, we make decisions based on choices presented with a singular context in mind. The choices made with said context will lead down a particular development path that becomes a part of “the architecture”.

As wonderful as the human brain is, one thing is for sure; we cannot recollect every single detail that went into the making of a particular decision. The reason for some decisions transcend code. When your development process helps convey the most important reasons in a way that makes it easy to comback to, pick up and continue, it relieves developers of the cognitive load in thinking through various decisions as you go along. The weight of this cognitive load breeds anxiety and anxiety begets decisions made without confidence, and that is when people start to feel like shit. They start disliking the development experience and the bad decisions may result in the increase of general technical debt.

This way of development is also way more expensive. When people feel like shit at their jobs they leave. When people leave, you lose a chunk of time going through the hiring process. Getting new people up to speed on a current project is a very expensive process, especially if the current process has knowledge gaps. New hires will have to constantly ask questions and depending on the number of questions they have and how early they get answered, it is all time taken away from immediate productivity. When you create a development process that scales, all things sort of fall in place and in the long term, it becomes very easy to optimize your app to scale. It also way less expensive in terms of resources including development time, down time and time spent hiring new people. All of this translates to money for a business but most importantly, people are happy at work because they will enjoy it.

So what do I mean by creating a development process that scales.

Before I go on to explain what I mean by a “development process that scales”, I want to establish the main focus of this post.

The main focus of this series of posts is to inform and advise developers on what to do at the early stages of their development cycle. From starting the project to just before there is more than 1 person working on it. That is not to say this cannot be applied to you if you have already started a project. You should bear in mind that as time goes on. It becomes more expensive and difficult to change course. But if you think you can absorb the cost of the change in your existing process, you are most welcome to do so. All the comments and advice in this post should be taken with your specific context in consideration because context is a huge factor in what makes the same decision right or wrong for 2 similar use cases.

When you build version 1 of your app to handle 5000 requests per second, that is not what I term a scalable development process.

Here are a few characteristics of a development process that scales. How these characteristics apply to your project signifies how scalable your development process is. I dont view scalability of the development process as a binary parameter. I view the scalability of your process as a spectrum when your trying to optimize your process to increase the scalability parameter.

When new team members are added to the project, the where to find resources, how to change resources and what resources are doing is easy. The time it takes for new team members to start making meaningful contributions to the whole project remains inversely proportional to the scalability of your development process. Because developers are expensive and the less time they spend understanding the system, the earlier you can get your monies worth. Yes, there are systems so huge that it takes months to understand the whole thing before you can make a contribution. But that is outside the scope of this post.
The next major telling factor in how scalable your process is how easily you can refactor and deploy changes without worrying about issues that regress? Are you confident enough in your changes that you can deploy on a friday night? This is what I call the friday deploy confidence factor. If the last thing you do on a friday before you go out for drinks with friends is the deployment of a new feature, how confident are you that it will stick. The anxiety deployment comes with is bad enough, and to deploy on a friday makes it worse enough. No matter how high your friday deploy confidence is, please dont deploy on a friday. Go out, have some drinks and enjoy life.
Another telling factor which determines how scalable your process is, is how easily you can switch out a component of your app and replace it with equivalent systems and trust that it will all come together nicely. A typical example is Payments. If your business succeeds in negotiating better payment terms with a different payment partner, how quickly can you switch to the new systems, knowing very well that the APIs will be different. The way you answer this tells alot about how your entire system comes together. Other components that you should be able to switch out include, loggers, caching layers amongst a few.
If you have a very scalable development process, one thing that tells is how much confidence you have in letting juniors contribute to an entire system knowing that you have processes in place to catch them when they mess up. Juniors mess up up. It is the God given right to. It is the only way they will learn. For a lot of people without a scalable development process, they give juniors mundane tasks to perform and slowly watch as they work, to gauge their readiness to take on the big stuff. If you have to watch them perform before you give them meaningful tasks, you are most likely doing something wrong. This is because you watch them because you are sure if they goof up, your systems will most likely come crushing. and the trust element is put in the human beings. But humans err, and if you dont have checks to catch potential errs, someone will.
When mistakes are made, how early are we in noticing, replicating, understanding, fixing and deploying patches is a key factor in how our development process scales. If a single request crashes your app, does the whole system come down with it? Do you have systems in place that notify you of crashes before users report them? In this particular case, being proactive prevents being caught by surprise and having to drop all you are doing unexpectedly. Also the longer you stay down, the more money you lose.

This list is by no means exhaustive, but one idea I am trying to put across is that your whole process should be one of proactiveness rather than a reactive one. Being proactive gives you the advantage of planning out how you approach issues. When you are reactive, you do things based on issues that spring up. When you are proactive, you are able to determine the cost for each process and execute in a manner that makes it enjoyable.

Essentially this is a series on things that reduce the cognitive load on developers, that way they have increased confidence in what they are working on and can deliver faster. In the end you should employ strategies that alleviate the cognitive load on your developers.

Now that we know what it means to have a development process that scales, we can talk about some strategies you can use to increase the scalability of your development process in no particular order.

Thanks to Chiamaka Nwolisa and Andrew Smith going through this article and suggesting changes.

The state of payment apps in Ghana 2019

Thu, 13 Jun 2019 00:00:00 GMT

The year is 2019. But once every month, someone somewhere in Ghana builds and releases yet another payment platform, because the existing ones dont work.

This has been the case for the past seven or so years; yet one sure bet I can place in this tech eco-system is that we will see a new payment system before the year ends.

I have implemented payment systems into apps in the past. They are all a pain to work with. Inbetween outdated documentations, unreasonable prices and badly designed error messages, mechanisms and fallbacks, I am yet to work with one that just works. But it’s simple to assume that if we know what is wrong we could build one that solves these problems. If it was that simple, some smart people will have already figured it out.

Here is what I think. Payment is broken and the sprouting of new payment services is symptomatic to this broken system. For a very long time I have spoken to people who build payment platforms to try to understand the particular problem they are trying to solve. I am yet to see anyone tell me something different.

Last Saturday, after the ConnectAndAccelerate meetup, I had a chat with @thedumbtechguy who is building a payment platform, @tonymobster who works in payment infrastucture, @silentworks @tsatsu @sir_krymx over the issue of payment in Ghana. My resolution is that payment methods around the world that already exist are fundamentally flawed, and the lack of proper governace makes it worse.

What I realised was a lot of people who venture into payment, take the problems they see at face value, without really understanding the problem before they start, and by the time they realise these issues, they are already in too deep to stop because they have some traction. They also give up on the ideals they started with because they realize the problems they seek to solve are symptoms of problems bigger than their entire being and the real solutions to those problems are bigger than their ideals and their existence. To actually solve the problems, they would have to get into the politics and governace of financial technology. This is not something you can do by writing code. To do this successfully, you will have to become a politician. Success for them will mean the investment of a lot of man hours and dollars without the return of any profits in a venture that they least know something about. But these are mostly startups and “Cash is king”.

I dont know how we can fix payments but what I know is, starting a new one is not the solution. I think the real solution will have to be tackled with politics at a governmental level. When that is solved, it will open up the solution to all the already existing platforms and they will instantly become good. But people who start payments system are not the kind who easily get into politics.

Unless there is a change to our approach to this solution, best believe, we are all at the mercy of the best of the broken payment systems.

Do you have any other theories or belive different, let me know on twitter

Golang style error handling pattern in Javascript

Sat, 04 May 2019 00:00:00 GMT

Updates

delventhalz mentioned on reddit that you can also easily achieve the same thing with

const res = await fetch('https://example.com/api')
    .catch(err => // Do error stuff);

I admit. Its a far easier, less messy way to achieve the same results, with less boilerplate. I think i need to go refactor my code now.

When I first learnt there was no try-catch in golang, I was sceptical. What could be better than a simple try-catch block. I soon realized that what golang had was much better for many reasons. In other for me to put my point across, let’s first explore the anatomy of a typical try-catch block.

A simple try-catch block takes this form.

try {
  /*
   * a block of code that is expected to throw an exception.
   */
} catch(ex: ExceptionType) {
  /*
   * handle the exception that you are expecting
   */
} finally() {
  /*
   * release resources that we held before exception was thrown.
   */
}

The first part of a try-catch is the try block. Most of the expressions in a try block is used to acquire a volatile resource that is expected to throw an exception. When these exceptions are thrown, you handle them inside the catch block, which is the second part of the try-catch block. Once an exception has been handled in a catch block, the finally block of the try-catch block preforms all operations necessary to cover, close and or terminate resources (to avoid leak). Most people omit the finally block entirely, and go on to perform resource terminations after the entire try-catch block has completed execution. I am not entirely for or against this pattern, because there are cases where they make sense and there are cases where they dont.

Most typical try blocks contain more than one expression that acquire different types of volatile resources. Which means if you have two expression of that nature, throwing exceptions of different types, you might need to account for the types and handle them differently inside the catch block. But different programming languages help handle this differently.

The main difference is between strongly typed and weakly typed languages. With strongly typed languages you can specify the type of the exception to catch and the rest is taken care of.

In strongly-typed language like Java you can have multiple catch blocks like this:

try {
    Obj1 obj1 = GetObj1(); // might throw Exception1
    Obj2 obj2 = GetObj2(obj1) // might throw Exception2
    // do something with obj2
} catch(Exception1 ex) {
    // do whatever with ex of type Exception 1
} catch(Exception1 ex) {
    // do whatever with ex of type Exception 2
}

You can even go as far as condensing multiple catch block into one like this:

...
  catch(Exception1 | Exception 2 ex) {
    ...
  }

If you have different ways of handling the different exceptions you will have to check the type of the exception. So your catch block might end up like this:

  ....
  catch(Exception1 | Exception 2 ex) {
    if(ex instanceOf Exception1){
    // handle ex of type Exception1
    } else if (ex instanceOf Exception2) {
    // handle ex of type Excetpion2
    }
  }

Handling exceptions in weakly typed languages like JavaScript is closer to the catch block condensation above. Because there are no explicit types, you are forced to type check.

The main problem with this is how you don’t know which expressions are throwing what exceptions.

If you run the program and an exception is not thrown, how do you know what can throw? The easy solution here is to add comments. But if you truely believe good code must explain itself, then this might not be an ideal answer.

Another solution would be to have individual try-catch blocks for each throwable expression. Doing this litters your code with numerous try-catch blocks. This makes your code difficult to read and maintain. It will definitely not be beautiful.

Another issue with this method is it relegates error handling to the bottom but this is a crucial part of the code and must be treated of the utmost importance.

I think error handling at the bottom of a function should be labeled an anti-pattern.

The third issue is the mental toll it takes to understand this block of code.

When reading code and you see a try block, you instinctively know there is a catch block but you dont see that next. So you make a mental note to look at what the error handling case is. If the try block becomes long or easily meanders, you can easily loose that train of thought.

If you are writing the code, the intuition is to write the catch block first and then deal with the try block. But this is not how you read code. If you read code like this, once you loose “the catch” train of thoughts, it makes the code more difficult to understand.

Golang avoids these three problems entirely by handling errors/exceptions atomically. To put it simply, it is a more lovely form of the multiple try-catch block answer stated above. A typical error handling in golang looks like this.

res, err := http.Get("http://example.com/api")
if err != nil {
  // handle `err`
}
// do something with res

The first thing this pattern establishes is you know which singular expression can error out, and you know exactly how that is dealt with.

Secondly, you are forced to think about errors as a first class citizen. You are assured that once something below this line is executed, everything above is fine.

Thirdly, if typical try catch block composition involve handling exceptions first, why don’t we entrench that pattern? This prevents the mental overhead necessary to keep track of what can throw. The mental impact of this is low. You get an error, you deal with it, you move on. Now, you can do other things.

This might not seem like massive gains but as your codebase grows in size and complexity, you realize that these little wins compound to make the codebase easier to reason about and maintain.

My job these days at AF Radio involves writing a ton of JavaScript and I set out to create a pattern like this in our codebase.

In our code base, most of the expressions that are expected to throw exceptions are Promises and rightly so. JS helps with the catch block of a promise. A typical promise execution looks like this:

fetch('https://example.com/api')
  .then(res => {
    // do something with `res`
  })
  .catch(err => {
    // handle `err`
  });

I’m not a fan of this method for promise execution. It forces you to implement logic in then-blocks while making it difficult to read. This method also exhibits the three issues listed above.

I prefer async-await, as they are much cleaner and easier to read. The fetch function above as an async-await will look like this.

const res = await fetch('https://example.com/api');
// do something with res.

When we wrap it in a try-catch it looks like this.

try {
  const res = await fetch('https://example.com/api');
  // do something with res.
} catch (ex) {
  // handle exception here
}

In other to turn this try-catch expression into the golang error handling pattern I mentioned earlier, I implemented a custom fetch function.

async function fetchable(url) {
  try {
    let res = await fetch(url);
    return [null, res];
  } catch (err) {
    return [err, null];
  }
}

To use this function we can easily call it like this:

const [ex, res] = await fetchable('https://example.com/api');
if (ex) {
  // handle exception.
}
// do something with res and exit

What happens is, one of two objects are always returned and never both. If the ex is truthy, the res object will not exist, and vice versa. If the underlying function throws, it will be captured in ex.

You can easily extend the fetchable function to take in more arguments.

You can also implement a utility function which takes a promise as an argument and returns a golang error handling pattern like this:

async function Executable(promise) {
  try {
    let res = await promise();
    return [null, res];
  } catch (err) {
    return [err, null];
  }
}

You can use this function multiple times like this:

const [ex1, res] = await Executable(fetch('https://example.com/api'));
if (ex1) {
  // handle exception.
}
// do something with res

const [ex2, data] = await Executable(db.query('SELECT * FROM USERS'));
if (ex2) {
  // handle exception.
}
// do something with data

This approach to error handling makes it very easy to understand the different failure points in your code and handle them appropriately without relegating it to some obscure levels in your code.

Let me know what you think. You can reach me on twitter @KiddBubu.

Thanks to Chiamaka Nwolisa for reading through with her super editing powers.

I moved my blog from Medium, here is why

Fri, 29 Mar 2019 00:00:00 GMT

I decided last month to move my personal blog from Medium to my own hosted solution called 5 Error using Jekyll and Github Pages. In this post I try to outline the very reason for my decision.

When I first got introduced to Medium in 2016, it was love at first sight. The idea was simple. Write your stories using the best web editor I had experienced as yet, and read the thoughts of others through their discovery engine. Nothing complicated. I wasnt the only one who fell in love. 1.9 million posts were created on Medium at the end of 2015. By the end of 2016 that number had trippled. And with that came a new funding round of $50 million in just a year after their last one. And that is where I think a lot of their issues started.

In other to explain where the problems started I will need to explain how VC funding works. But let me leave that intricate explanation for another post. In summary, medium needed to generate some form of revenue, and they had chosen not to do it with ads. That left very few options on the table. Of which a subscription model of some sort was one of them. Working with creators becomes a little bit tricky especially when you build a content platform based on subscriptions where none of the proceeds go to the content creators. Imagine spotify wasnt paying royalties to their artists. That might open a pandora’s box of legal issues. In this particular case, the creatives on Medium become the writers. Medium decided to help this issue by paying out some cash to writers depending on how popular a story got. That is ok. But how they decide to do it is what makes it difficult for me.

I am a part of the new generation of Africans who are trying to build a tech businesses in the hopes of becoming our very own unicorns. That is not without its challenges, especially being part of the first wave. On large parts, we get through it by sharing ideas. For those of us who can meet up, we talk about our shared challenges and for those who cant, we write about it and hope others read it. For a very long time the internet has been a place where we could easily find content that could help us go through a lot of the challenges we face. But when the experience of creating startups on the internet is largely filled by people from Europe and the US, it misses a lot of context that has to do with our particular environment. An easy advice will be, well, get more Africans to write on their experiences in building startups. I will tell you its not as easy as it seems. For those of us who can write about it, we try our best and for a very long time medium was a popular choice for writing.

For a very long time Medium was the place where I could easily dicover content from other Africans, people wrote and depending and how good a piece was and the claps/hearts it got, Medium would pick them and share them. So you never had to worry about how good a story was, if it was on your medium homepage.

A few weeks ago I had an experience with a startup that left me feeling terrible about how people who wanted to do startups didnt exactly know how to go about building their products. In my bid to share some best practices based on my experiences, I wrote this article and my intention was to share it with other Africans. It got popular and very soon I was invited to add my piece to the Noteworty publication on medium. I though it was good. More eyeballs for my article. Note long after, when I opened my article, there was a notification at the top my article that said my article was eligible for the metered payed wall.

I didnt really know exactly what that meant or the implications of it. I went on to learn more. The two benefits to the metered paywall were clear to me; to earn money and for stories to be put through the recommendation engine that powers medium. That included the medium homepage, their topics page, their daily digests and their apps. The sticking point for me was what happened when you opted out; only your medium followers could discover your story. Now for someone like me who had a low follower count on Medium, getting my story into the recomendation engine was kinda a big deal for me, especially once I knew it was possible. So I decied to join the partnership program so that at least people who were presented with the story would either view it in private mode on the browser or figure out another way to outsmart the Medium system. Then I faced a new set of challenges.

To join the partnership program, I had to have a Stripe account. In other to have a stripe account, you need to be in a country where stripe was available of which my country was not a part. If you really want a stripe account you could get it by joining their Stripe Atlas program at a cost of $500. If you ask me, this wasnt worth it, especially if you are not a business. If you think about it, it means that, if you didnt have Stripe in your country, nobody could ever dicover your stories on Medium. This means the recomendation engine was limited to people in just 32 countries where Stripe is available on some level. If you have gotten this far in this article and are African, please start to wonder why you have never recently seen stories from other Africans who are not in your network. Yes you guessed it. Stripe is not available anywhere in Africa.

I am sure Medium made this decision to include stripe as the only payment partner based on valid business decissions. I strongly belive in capitalism and its power to alleviate poverty especially in my part of the world, but when decisions like these go to affect the content that I can see on Medium, then I think the time has come to take a stance against it. For people who are out there looking for advice and inspiration on how to build their businesses in other African countries, its quite unfortunate the only quality stories they are going to be getting are ones of very little to no value to them and even sometime the wrong one. Others like Yann Girard might think the partnership program is cool and benefitial to people in Thailand, but when the people of Thailand can not have a Stripe account, I struggle to see how this goes to benefit them. If you are out there and are still sticking to Medium, all i can say is, just be careful what you read. Because as much as they might align to your interests, there might not be much diversity to the content after all.

R.I.P Medium. It was good while it lasted.

Thanks to Chiamaka Nwolisa for reading through and catching issues.

How to not build your application

Sat, 02 Mar 2019 00:00:00 GMT

Every now and then i get asked by brilliant people to help them test their products, and most of the time I leave those encounter feeling less enthusiastic about using the product or service than I was before I came into the meeting. Yes your idea is good, I want to use your product but does your product do what you say it does? Most of the time it doesn’t. Here are a few tips on what to do and what not to to do for creating your first product based on my short experience doing so.

Build Version 1 before Version 100

As simple as this statement is, a lot of people make this critical mistake. People want to release a product that is fully baked as an initial version. This leads to a lot of assumptions about what users want, leading to features, nobody needs. I think the first version of your product should do one thing and one thing only; doing exactly what your value proposition says. If your value proposition is helping people easily file tax returns to save them time and money, you have no business building a feature that shows them how they spent their money over a year. When you are successful in helping people properly file their tax returns, then you can take up other features based on the usage patterns.

Version 1 shouldn’t take longer than 1 month to build.

The first iteration of you product should never take more than a month to build even if its one man doing all the work. If it does, you are probably doing it wrong. Unless you are building yet another payment platform, then you have your work cut out for you. But there are 5001 other payment platforms, what are you building that the others have not. I digress. This point ties itself to the initial point i made. You initial application should not do anything beyond what your value proposition says. If it does, perharp, you don’t properly understand what you are doing in the market you are in.

Listen to your users. But don’t build features they request.

It’s really hard to design products by focus groups. A lot of times, people don’t know what they want until you show it to them. — Steve Jobs

If there is one thing users are good for it is identifying what the problem is. But when it comes to solutions, they are the last people you should listen to. This is because the kind of solutions users propose is biased towards the experience they have, and lack the nuance required to optimally solve a particular problem of this nature. I experienced this when I built an inventory system for a client some time ago. On the page on which I listed all his inventory, the content on the page was paginated. It was a no brainer. But he called me later and demanded the pagination be removed and instead, have about all 100 plus listed on a single page. I asked why he wanted it that way and he told me so that when he scrolls to the bottom of the screen he can know exactly how many items he has based on the number of the last item. I just easily added this number to top of the screen and he admitted it was a better solution.

Outsource all the parts of your app you can to services.

Part of getting your product out the door within the first month is knowing exaclty what to build and what not to build. Saying you don’t trust a particular service is a sign of amateur hour. Use cloudinary to handle file uploads. Remove the complications with setting up an s3 bucket, keys and the unnecessary complications you will have to deal with.

You have no business implementing search in your app. Use Algolia. if you dont, chances are you are going to do it wrong and it can cripple your app. Yes search is that difficult. Do you need login, use Facebook Login, Github login, LinkedIn Login, even Snapchat login. One less password to remember is good for the planet.

Use UI frameworks

When i talk to people, in their bid to build something that looks unique, they spend eons writing styling rules that don’t have any bearing on the solution they are about to offer. Dont spend anytime on figuring out the right shade of blue. It doesnt have to be too pretty. It just has to be functional. Having a unique design is not part of your value proposition. If it is, you are doing it wrong. What happens when the next person out designs you. It is for this reason ui frameworks were created. They contain commonly used components and you can easily come up with a beautiful product just by putting together a few of these components. Some popular ui frameworks include semactic-ui, bootstrap, UI-kit and Cube UI. Some people on the internet took their time to put them together. Take advantage of them and thank them.

If the answer to why you are building a feature starts with “I think the user would like to…”, dont build it.

Even when you build solutions to solve your own problems, the time you spend on your project makes you loose sight of everything else in your peripherials. You are then no more the user. You dont know what the user thinks. Get your product out the door and let the real user decide if what you think the user is thinking is actually what the user is thinking.

You dont have scaling issues. Dont build to scale.

Another common theme amongst first time product creators is the misconception that you need to build with scaling in mind. Scaling is an issue you cannot simply build for when you have never had the experience. If you are not google your initial product will not get to the intial scale your are thinking. Thinking this way only leads to complicating what your are building, which might sometimes result in your app never making it out the door in the first place. All want to achieve scale. Few people achieve scale, and when you do thank the stars. But by the time you do you will have the resources necessary to solve your scaling issues, and even if you don’t, there will be people willing to throw resources your way to solve this problem.

Tech is not your solution. Tech scales your solution.

After evaluating the problem you are going to solve and deciding what you are going to build, most first time founders get easily confused about the role of tech in the business they are going to try to build. Unless you business is an encryption algorithm, facial recognition software or a machine learning module built on the blockchain that takes advantage of sever-less cloud infrastructure then there is a good chance your solution can stand without the tech. How do you know this? Think through you product and you will realise that without tech you can also provide the same solutions as you would with the tech, though it might be slow and painful. Eg. Payments. The basics of payments involves taking money from one person and giving it to the other. If you make the physical journeys to fulfil this request, you are solving the same problems. This is only one example but I implore you to think about what your solution is and how you are using tech to optimise the delivery of that solution. This helps you understand which part of our product to abstract with doing things that dont scale.

Track everything, and create funnels from your tracking data

In my opinion this is the only way through which you should build features. Everything that can be clicked on should be tracked, along with contextual data. Mixpanel is a good place that can help you start this. Events in google analytics is also a good. Tracking events in your app should be in place from day 1. This will help you figure out the most performed actions, the number of steps people take to perform particular actions and where people are dropping off when they are performing an action. This informs you on how you can easily optimise for frequently used features and make you app better.

This is by no means an exhaustive or holistic approach to building the first version of your applications, but this is a very good starting point. Take into consideration what you are building and for who, and let it influence your decisions. At the end of the day all I am saying is, try to significantly reduce the amount of work you need to do to get your app out the door, because getting it out is the only true litmus test of your idea.

If you have any other thoughts or ideas on this subject please feel free to share them on twitter. You can find me @kiddbubu

Thanks to Andrew Smith and Yaw Boakye all of DevCongress for reading through this and suggesting edits.

Take a step back. Grow up for a second.

Thu, 06 Oct 2016 00:00:00 GMT

If you aspire to be a software entrepreneur, you have a lot to learn. The more I learnt and looked ahead, the more I realized that I knew nothing.

In the words of Einstein,

‘’The more you learn, the less you realize you know.’’

When the world around you notices how much you have grown from all the learning, it wants to tap into you. What I mean by that is a whole different post so let’s leave that for another day.

I can’t speak for those on this path with me, but when I look back at myself about 6 months ago, I understand now that I was blind.

As entrepreneurs, we want to solve problems, and make money in the process. This isn’t often the case but that is our high. Solutions come to us as instantly as we see problems. Validation is just one of the boxes you have to check off. You pray there is someone out there who feels the pain that you believe to have identified, will log onto your website, put in their card details and click confirm. You dream of someone who will download you app never to uninstall it. When speaking with potential customers and users to find better understand their pain, we are careful not to impose a potential solution. We have learnt to isolate our very own ideas of what we think the solution is or should look like. With time and practice, we muster the courage to fight the urge to ask this very question every single day:

“I have an app that solves [this problem you are having]. Will you use it?”

Every so often when we spot a problem, most times our immediate solution sounds something like this,

“An app that helps you [envisioned solution goes here]”

But let’s take a step back, and grow up for a second. I say grow up simply because grown ups are associated with being in the habit of thinking through their problem, though it’s not always the case. So in the spirit of reasoning, let’s grow up for a second and ask ourselves,

“Is the solution to a problem necessarily an app?”.

When we disregard the impact of this question we end up building “new straws that help us drink water easily”. Sometimes the simplest answer to some of these problem is for the human race to grow up. But asking that is asking a fish to climb a tree. We have never been more stupid — so we get apps to abate our stupidity.

Let’s face facts. Its 2016. Apps are the new rage. There is an app for everything. From finding sex, to mapping all the places you have pooped at (yes, that actually exists). There is always that app that promises the ultimate solution. Do we really need to build one more app that will forever chase home screen real estate? I dare ask once again, do I really need an app to solve this problem?

We need to start rethinking our approach to solutions as a whole. It is one thing to identify a problem. It is another thing to identify a solution. It is an entirely different thing to find a solution that optimally solves the problem you identified. It might not seem like much, but it dictates how you will go about solving your problem. It is for this same reason that there are hundreds of ‘Todo’ apps out there, but we all still love one in particular, despite all the ‘blissful’ functionalities it may lack. (I love Google keep).

If you catch yourself thinking that building an app is the solution even before you have thought through why people are having a particular problem, perhaps you need to take a step back and grow up.

I hope we learn to really analyze the solutions we create to solve problems. In the meantime, I pray that Chanl isn’t a straw that ‘bends at the elbow’ and helps you drink water easily as you are lying in your bed.

The kenkey attitude

Thu, 06 Oct 2016 00:00:00 GMT

I am a Ghanaian. I love kenkey. But most foreigners can’t seem to grasp the fact that we love food that is fermented.

Let me give you a brief about kenkey.

Kenkey is a Ghanaian staple made from maize. The maize from the husk is dried, then soaked in water for a few days, and the milled into maize dough.

To make really good kenkey, the maize is left to ferment. for about 3 days. This fermented dough is also used to make a lot of good foods like my personal favorite, Banku (with Grilled tilapia).

To the people of Kenya who have a lot of meals made from unfermented maize dough, this process seems strange. The fermentation part i mean.

If you still remember your high school science, the fermentation process turns sugar into alcohol, in this case the starch in maize is turned to booze.

So kenkey for lunch on a work day simple means u get to drink on the job.

Cheers :)

Open sourcing Ambre

Wed, 06 Apr 2016 00:00:00 GMT

The MEST journey is a rather intense one. In MEST growth is inevitable — on the personal, professional and any other level. I don’t think I have ever changed as drastically as I have within this one year experience.

Capstones at MEST are projects during which participants come up with solutions to existing problems in the form of a good mix of technology and business. It is an exercise to train our software entrepreneurial minds. One of those capstones led to the birth of an idea for that solved a problem I experienced first hand.

While Philip and I were in the process of building Chanl, we got to a point at which we had to incorporate a system to collect feedback from users remotely. I searched, but the solutions I found were lacking in one way or another. So we built our own solution and called it ‘Autofeedback’. Philip would build the Meteor backend/dashboard and I would build the Android library. It was basic, and it solved our problem. Later on, Nyawira, Edmond and I picked it up for our very first capstone at MEST — to find out if we could build a business around it. Nyawira would spearhead our new found cause as the business lead while Edmond and I would see how we could improve the product. We called this new venture Ambre.

Ambre was a feedback aggregation engine that gathered feedback from all platforms on which a product was mentioned. From social media, email and in-app widgets, Ambre would collect out all your feedback for you.

We added a few more features to it such as the ability for mobile app users to shake their phones so as to capture the screen, draw on the part with an error/bug and attach a message to send to the developers. Ambre also carried out sentiment analysis on the feedback aggregated and indicates the level of positivity/negativity that your app users held towards your product. Finally, you could gather feedback in the form of reviews and ratings on the application stores like Google PlayStore and possibly others.

Even though we couldn’t add all the features we intended to, it was wholly successful. In our newfound pursuit, we discovered startups and other businesses that were thriving at solving this problem like Doorbelland Instabug. In order to keep Ambre alive, we would have had to dedicate a whole lot more resources to it — resources that we simply didn’t have at that time.

So we packed up Ambre. We still used it in Chanl though. Fast forward a few months to today and Ambre is still packed in a private repository but, as Nyawira would say, she’s wonderful. For that simple reason, I am open sourcing Ambre. Anybody who is interested should look at it and if they can improve it for the world, just submit a pull request. Click hereto view Ambre on Github.

In the meantime, get Chanl and begin experiencing tangle-free group chats.

Available for download here: http://j.mp/chanl

The Idea

Tue, 12 Jan 2016 00:00:00 GMT

Once upon a time, I was a college student whose departments used WhatsApp groups for the dissemination of information. These groups became extremely useful during examination periods because exams times could be changed within a moment’s notice. Being the class prefect’s duty to inform all students of these changes, what better way to do so than to post a single message in these designated WhatsApp groups. These group chats consisted all members of the class and unfortunately, would be used by disgruntled students to express their displeasure at the changes.

This meant two things: Being offline for a while would result in over 500 unread messages awaiting your return. The constant dilemma: You either sift through the several unread messages looking for probable announcements you missed out on or choose to ignore these unread messages with your fingers crossed, hoping no significant changes had been made to your exam/class schedules.

Obviously, none of the options from the dilemma were favorable for a student trying to study for an exam. Therefore, I set out to build a solution to a problem that I was definitely not experiencing alone.

‘Chanl’ was born.

I have spent the last year tweaking, turning, typing, changing, deleting, restarting and piecing together fragments of code hoping to create a viable prototype. It wasn’t until I shared my thoughts on Chanl with Philip that it became evident how wrong my approach had been. Philip was soon to be my co-founder and CTO, but most of all my friend and partner in crime. With him joining my mission, we set about to plan and execute this app and make it the seamless experience you shall hopefully enjoy.

Define happiness

Sun, 03 Jan 2016 00:00:00 GMT

There is a small village in the eastern part of Ghana. On the outskirts of Hohoe in the Volta region. It’s called Wli. This town is home to about a thousand people, whose main source of livelihood is farming. This village is also home to the highest waterfall in West Africa with a height of about 143 meters from the top of the upper falls to the pool into which it pours. So there are a lot of tourist who visit this village to see the waterfall in all of its glory.

I happened to be one of the tourist visiting the waterfall, as part of a road trip my friends and I went on towards the end of 2015. When we got to the town mid afternoon, the town looked dead, with not a single soul to be seen walking the one road that went through the town, except for the visitor center that was at the entrance of the forest of the waterfall. Wli was our final destination for that day, so we dropped our backpacks and camped. The weather was magical.

After dinner that evening, we heard some drumming and singing in the town. We decided to go see what was happening.

On the streets that looked dead a few hours ago, the youth of the village had turned up. Chanting local songs, with drums, they sang, danced and marched through the streets. Those who couldn’t join the march, cheered them and quenched their thirst with water and bottles of Kasapreko. There was no occasion. This was how they jammed through the nights. There was no segregation, none to old, none too young, none too white, non too black, none too local, none too foreign. As long as you could turn up, you were a part of the celebration. We turned up with them too. They shared a communal love for each other. There was a mother who even tied her children together on a bench. Those around watched on for her while she danced for about half an hour, came back, said her thanks and offed with her children. The Wli people defined their happiness. They were happy.

It took very little to make the people of Wli happy. They didn’t need no 3g or electricity. They didn’t need to tell the whole world what was happening on Facebook. They didn’t need to let the world know what was on their minds with Twitter. No need to share pictures of what was happening on Instagram. Here was a community of peasant farmers, happier in one night with none of the modern day techno garbage, than some of us will ever be with all our cool tech in our entire lives.

Let me let you in on a bit of my background. I was born and bred in the big cities of Ghana and was fortunate enough to attend one the best schools the country has to offer. I have a degree in computer engineering and am currently in training to become a software entrepreneur at MEST; to ‘build a globally successful software company’. I haven’t known abject poverty or extreme riches. My parents have provided enough, to make living comfortable. I have never lacked.

My parents always wanted me to get a good education, so that i could get a good job with a good salary. What does it matter if it doesn’t make me happy. After all, I will have money. The rest can follow later. I spend my days working, and am so tired by nightfall to do anything else. I wake up the following day and the cycle continues.

Of late I have began to question everything. We wake up everyday chasing a dream, we hardly ever catch. And for those of us who do, it is often without people with whom we could share it. Because the tunnel vision we often develop to catch our dream often ignores the people who really matter to us because they fall into the peripherals. By the time we realize it wasn’t all worth it, it is often too late. But we have heard the stories and we know most of the time, we are chasing a mirage. We are blinded with vanity. And for people who get out of the vain chase in search of what makes them happy, they are met with people who look on bewildered asking ‘what the f*ck are you doing mehn?’. We live our lives generally bitter about everything and hatful towards people.

The declaration of independence states that

“…We hold these truths to be self-evident, that all men are created equal, that they are endowed by their Creator with certain unalienable Rights, that among these are Life, Liberty and the pursuit of Happiness…”

We equate money to happiness and often think our cool modern toys will make us happy. We wake up daily to jobs we hate because the money it good, in hopes that it will make us happy. If we ever find out what really make us happy and are ever brave enough to do them, we will become better people generally. The world will even have a chance of becoming a better place. But it is not easy to wake up one day and just quit your day jobs. It requires very long term planning and and a lot of courage. Did I mention long term planning? The system has trapped us. Its not our fault. If you ever find out you can quit before it too late, take a chance if you have the courage. It is worth it.

Take a leaf from the book of the Wli people.

Define happiness.

Try to live happily ever after.

Trash Apps from Africa

Wed, 23 Dec 2015 00:00:00 GMT

The world has gone through a paradigm shift. A shift in which mobile phones have become the focal point of our very existence. Everything you can think of and want to do, there is an app for it. Everything! From sex to learning new languages and everything in between. Despite our many problems, Africa has not been left behind in this shift. In fact some may say Africa is ahead of this shift. In Africa, everybody has a device, that can at least, make a phone call and receive text messages. The rate at which we have adopted mobile phones in our daily lives is astonishing. This growing rate has shown no signs of slowing down. I doubt it will. Mobile phones have become a utility for all rather than a luxury for the affluent in society. I can not over emphasize how much this has gone to improve lives. Because that is common knowledge.

The Adoption of smartphones in the part of the world has gone through the roof. everybody wants Whatsapp. Personally, I think that is the single most motivating factor for people in this part to get a smart phone. Eventually people will migrate from the yams (feature phones as we call it here). I’ve never been so sure about the future than this. People also want Facebook. and all the cool apps various app stores have to offer. Twitter, Vine, Instagram, Facebook to say the least.

All this makes me feel good. In this direction, Africa is moving forward, and a tremendous speed at that also. A direction i am glad to be a part of. But there is one question that nags me; How many home made apps are used by Africans?.

This is not a cry for Africa to adopt things made African. NO!. That is a stupid cry. I will not pretend to be a saintly African evangelist who has come to preach to you goodness, righteousness and how glad it will make your heart along with the salvation it will bring to your soul if you adopt the trash our ecosystem has generated, which has the capacity to run on a snapdragon processor and can be started by pressing a hideos looking hieroglyphic on your mobile screen. Some may even call it an app. Never!. I will not pretend to not be guilty in that respect as i very much am. The only African made apps i have on my phone are Slydepay, a payment app, i have never used and one i downloaded because, i’m friends with Edmond who is one of the developers, and GHConstituion, my very first app on the playstore, one i keep because i built, and hardly ever use. I will not bat an eyelid to remove these apps to free up space on my device so i can install something western made. I think i already have, without remorse.

If you think some of what am saying are not true, just ask an African to remove Whatsapp, and if you leave with all your teeth intact, with only insults directed at your whole family, you are among the lucky ones.

I don’t think the problem of we not using African app has anything to do with the fact that they are made in Africa. People who use apps don’t give a f*ck where an app is made. Whatsapp can be made in Syria, Facebook in North Korea and instagram in Iraq for all they care. So far as it serves a purpose, people will gloss over the small details that don’t matter.

What a lot of developers in this part of the world miss is the keyword ‘purpose’ when making apps. People can download apps out of pity and the fact it was made by a fellow black brother on the continent. But those downloads can only go so far for so long. It’s worse if your app is intended to be a business. I’m sorry to disappoint you, close shop before you waste investor money if you have any, be it your own.

Mobile apps, have to be designed, to every detail, to the smallest intricate detail. From the way icons will look like on different screens; Amoled or LCD, how screens transition into each other, To even how the colors in the app make users feel. That is the very thing i think is lacking in the design of mobile apps in this part of the world. In truth no user will notice the fact that you used some complicated algorithm written by the brightest developer of the lot, to get and arrange some particular set of data when you UI looks like crap. Sure ur app might do whats its intended to do, bottom line is it still looks ugly. It can be written in Haskell, Scala, Clojure, OCaml, Lisp, Prolog or even Assembly, if it looks horrible, not even the devil can help you. It get worse when your complicated algorithm keeps crashing your app, you are assured of a definite uninstall. That, i wish i was lying about.

We claim to learn app development, when the only thing we are being taught is how to write code, which is the least important part of the whole process of app development. Where is the user experience. What happened to design. How is the business side?. All these might sound over the top but is the very core of an app and if you miss these part, you are doomed to fail. Code is just a small part of the big picture and the sooner we realize that, the sooner we will make apps that are worth keeping.