how to transfer oversized string over message system

for text based message system, if there's a limit of x KB, you can still transfer (~5 * x) KB by

1. compressing the original payload
2. encoding compressed binary to text
3. sending it

receiver reverses the process to get original payload by

1. testing to make sure message is encoded
2. decoding it to compressed binary
3. uncompressing decoded binary

import org.apache.commons.codec.binary.Base64;

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

String payload = """
        {
        }""";
System.out.printf("original size: %s bytes\n", payload.length());

ByteArrayOutputStream baos = new ByteArrayOutputStream();
GZIPOutputStream gos = new GZIPOutputStream(baos);
gos.write(payload.getBytes());
gos.close();
byte[] compressed = baos.toByteArray();
System.out.printf("compressed size: %s bytes\n", compressed.length);

String encoded = Base64.encodeBase64String(compressed);
System.out.printf("encoded size: %s bytes\n", encoded.length());

System.out.println("is message encoded? " + Base64.isBase64(encoded));
byte[] decoded = Base64.decodeBase64(encoded.getBytes());
System.out.printf("decoded size: %s bytes\n", decoded.length);

GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(decoded));
ByteArrayOutputStream output = new ByteArrayOutputStream();
byte[] buffer = new byte[4096];
int bytesRead;
while ((bytesRead = gis.read(buffer)) > 0) {
    output.write(buffer, 0, bytesRead);
}

gis.close();
String uncompressed = output.toString();
System.out.printf("uncompressed size: %s bytes\n", uncompressed.length());

original size: 9450 bytes

compressed size: 1235 bytes

encoded size: 1648 bytes

is message encoded? true

decoded size: 1235 bytes

uncompressed size: 9450 bytes

revisit timeout settings after 9 years

After almost 9 year, this is part 2, or an update to Are you setting Connect Timeout correctly? I'm going to give the best practice of setting connect timeout and read timeout for org.springframework.boot.web.client.RestTemplateBuilder as of Spring Boot 2.7.13.

Let's say it takes up to 0.1 sec to setup connection and 1 sec to receive response, then it's easy to understand you'll get org.apache.http.conn.ConnectTimeoutException if

return restTemplateBuilder

.rootUri(rootUri)

.basicAuthentication(username, password)

.setConnectTimeout(ofMillis(10))

.setReadTimeout(ofMillis(1000))

.build();

and java.net.SocketTimeoutException: Read timed out if

return restTemplateBuilder

.rootUri(rootUri)

.basicAuthentication(username, password)

.setConnectTimeout(ofMillis(100))

.setReadTimeout(ofMillis(100))

.build();

You can either set both timeouts

return restTemplateBuilder

.rootUri(rootUri)

.basicAuthentication(username, password)

.setConnectTimeout(ofMillis(100))

.setReadTimeout(ofMillis(1000))

.build();

or omit connect timeout and only set read timeout

return restTemplateBuilder

.rootUri(rootUri)

.basicAuthentication(username, password)

//.setConnectTimeout(ofMillis(100))

.setReadTimeout(ofMillis(1000))

.build();

however by setting connect timeout only and omitting read timeout, you'll get java.net.SocketTimeoutException: Read timed out

return restTemplateBuilder

.rootUri(rootUri)

.basicAuthentication(username, password)

.setConnectTimeout(ofMillis(100))

//.setReadTimeout(ofMillis(1000))

.build();

Let me rephrase what I said 9 years ago, don't set both timeout to be the same, which shows you don't know how network communication works.

mixing spring boot web and webflux

Following how to use reactor scheduler in micro-services, things becomes even more tricky if you mix spring boot web and webflux.

subscribeOn

[nio-8090-exec-1] -----Service : foo

[oundedElastic-1] EmailServiceClient : Sending email notification

[ctor-http-nio-2] EmailServiceClient : Sent email notification

publishOn

[nio-8090-exec-1] EmailServiceClient : Sending email notification

[nio-8090-exec-1] -----Service : foo

[oundedElastic-1] EmailServiceClient : Sent email notification

publishOn + subscribeOn

[nio-8090-exec-1] -----Service : foo

[oundedElastic-1] EmailServiceClient : Sending email notification

[oundedElastic-2] EmailServiceClient : Sent email notification

Mono.just + subscribeOn

[nio-8090-exec-1] EmailServiceClient : Sending email notification

[nio-8090-exec-1] -----Service : foo

[ctor-http-nio-2] EmailServiceClient : Sent email notification

Mono.just + publishOn

[nio-8090-exec-1] EmailServiceClient : Sending email notification

[nio-8090-exec-1] -----Service : foo

[ctor-http-nio-2] EmailServiceClient : Sent email notification

how to use reactor scheduler in micro-services

Depending on how many cores we have for your micro-services, we may have as low as 4 threads (reactor-http-nio- or reactor-http-epoll-) when it comes to listening and talking to outside. If these threads are blocked in a way we didn't expect, the performance could be worse than traditional synchronised paradigm. Here're a few examples to show how it works when offloading blocking operations to a larger thread pool at different places. Logs happen at doOnSubscribe and doOnSuccess.

no scheduler

[ctor-http-nio-4] Controller     : Received request

[ctor-http-nio-4] ServiceClient  : Getting something

[ctor-http-nio-4] ServiceClient  : Got something in 116ms

[ctor-http-nio-4] Controller     : Processed in 120ms

scheduler on subscribeOn WebClient call to 3rd party

[ctor-http-nio-3] Controller     : Received request

[    scheduler-3] ServiceClient  : Getting something

[ctor-http-nio-4] ServiceClient  : Got something in 116ms

[ctor-http-nio-4] Controller     : Processed in 120ms

scheduler on publishOn WebClient call to 3rd party

[ctor-http-nio-5] Controller     : Received request

[ctor-http-nio-5] ServiceClient  : Getting something

[   scheduler-12] ServiceClient  : Got something in 116ms

[   scheduler-12] Controller     : Processed in 120ms

scheduler on subscribeOn and publishOn WebClient call to 3rd party

[ctor-http-nio-3] Controller     : Received request

[    scheduler-3] ServiceClient  : Getting something

[    scheduler-4] ServiceClient  : Got something in 116ms

[    scheduler-4] Controller     : Processed in 120ms

Bonus cases to make things even tricky.

subscribeOn after doOnSubscribe in controller

[undedElastic-21] Controller     : Received request

[undedElastic-21] ServiceClient  : Getting something

[ctor-http-nio-7] ServiceClient  : Got something in 116ms

[ctor-http-nio-7] Controller     : Processed in 120ms

subscribeOn before doOnSubscribe in controller

[ctor-http-nio-3] Controller     : Received request

[undedElastic-21] ServiceClient  : Getting something

[ctor-http-nio-4] ServiceClient  : Got something in 116ms

[ctor-http-nio-4] Controller     : Processed in 120ms

What is this method doing?

One-liner method passes its parameters untouched to anther method doesn't usually add value. I came across this method the other day, and wanted to look into it. I know this is not a one-liner.
private String base64EncodedFrom(byte[] bytes) {

   if (bytes == null) {

       return null;

   }

   return Base64.encodeBase64String(bytes);

}

Sure it checks if bytes array is null beforehand, its name is a bit more human readable, because of the From, than the one that it'll pass on. To make it even better, do you think the method name should be something like base64EncodedFromByteArrayNullSafe? Well, that's not my point, if you've read this post.

Let's check Base64.java

public static String encodeBase64String(final byte[] binaryData) {

   return StringUtils.newStringUtf8(encodeBase64(binaryData, false));

}

public static byte[] encodeBase64(final byte[] binaryData, final boolean isChunked) {

   return encodeBase64(binaryData, isChunked, false);

}

public static byte[] encodeBase64(final byte[] binaryData, final boolean isChunked, final boolean urlSafe) {

   return encodeBase64(binaryData, isChunked, urlSafe, Integer.MAX_VALUE);

}

public static byte[] encodeBase64(final byte[] binaryData, final boolean isChunked, final boolean urlSafe, final int maxResultSize) {

   if (binaryData == null || binaryData.length == 0) {

       return binaryData;

   }

...

}

Now we know encodeBase64() is null-safe. Let's then check StringUtils.java

public static String newStringUtf8(final byte[] bytes) {

   return newString(bytes, Charsets.UTF_8);

}

private static String newString(final byte[] bytes, final Charset charset) {

   return bytes == null ? null : new String(bytes, charset);

}

So newStringUtf8() is null-safe too. Now it's safe to rewrite the first method this way
private String base64EncodedFrom(byte[] bytes) {

   return Base64.encodeBase64String(bytes);

}
Do you still think base64EncodedFrom() is a method that adds value?

30th Coding Anniversary

I was selected into a interest group learning programming in BASIC (using Apple IIe and Laser-310) in December 1985, when I was in year 8. It was fun. I learned coding so hard that I got first prize award (equal 1st, Year 7-9) in programming competition in my home city few months later in 1986. One question in the competition was swapping the values in 2 variables without using a 3rd variable. My name was published on newspaper and I told my parents I'm famous.

Admission ticket, 1986

I got three more first prize awards later, 4th place (Y7-9) in 1987, 3rd place (Y10-12) in 1988 and 2nd place (Y10-12) in 1989. The last one in June 4th 1989 was the most difficult. I got mumps below my left ear and had been stayed at home and almost couldn't eat anything for a week just before the competition, which I sit in No. 15 Middle School, very close to my then home.

First prize certificate, 1986

These are few things happened before I went to university studying computer science and software engineering. I have come a long way and these things always remind me of where I came from, and where the determined boy wanted to go three decades ago.

Merry Christmas and Happy Coding!

Don't mock microservices

Didn't I just post Use Caching Service as Mocked Microservices? Yes, that post is basically saying that instead of each test case/suite starts its own mocked HTTP server, sets expectation, sends request to it then shuts it down, as a side-product, a caching service can be use as an alternative.

The idea behind is, client calls dependency services in test environment in the same way it calls dependency servies in production environment. How about we go one step further, always call a real service instead of its mocked version?

This used to be hard for monolithic architecture. With shifting to microservices, it's just a matter of few more deployments, a service can provide test / staging / UAT environments to clients. This removes client's burden to set up and tear down test fixtures, so that test code can reuse production application context configuration, if you use Spring Framework. But that's not all.

The biggest benifit in my opinion is, it removes the grey area of responsibility between service provider and consumer. Who's job it is to update a mock service's behavior when the real service changes its behavior? Just imagine how many times your unit tests pass but still fail in integration test or production, simply because the real service now returns B, but your mocked service still returns A.

Better communication within team doesn't solve this problem, real-time communication between service consumer and provider does.

Use Caching Service as Mocked Microservices

For a long period of time, we've been using mocking framework to ease unit testing. The idea behind is,

• in production, object A's method a() calls object B's method b()
• in test, we
• pre-define b()'s behaviour
• inject mocked object B into object-under-test A
• fire a() and verify the output

With the shift from monolithic style to microservices oriented software design, we call remote APIs (RESTful or RPC) more often than we did before. Quite a few HttpMock products were created to allow us to test remote APIs the way we used to test local APIs.

But if we have a real HTTP server somewhere, in order to mock response from remote, all we need is to set up endpoint with the response and later tear it down. As a side-product, caching service is such a HTTP server that can easily be used to mock remote APIs object-under-test consume. It works this way

• in product, object A's method a() send HTTP request to service B's endpoint b
• in test, we
• pre-populate (POST) expected response in cache server for endpoint b
• send (GET) HTTP request to it (instead of the production endpoint) from A.a()
• get response from cache server and verify a()'s output
• remove (DELETE) the response from cache server

Now you see how the idea of microservices may also change the way you test your microservice oriented production code.

Happy testing!

Method names are new (redundant) comments

It's very interesting to see how developers react to commonly accepted coding conventions.

We all know methods should be short, and comments can be avoided when methods are short and with meaningful names. But by following these conventions, we sometimes only generate another form of redundant comments. Let me explain it with an example.

    if (feeType == FeeType.PERCENTAGE) {
        ...
    }

What's the problem here? It reads not like in human language, or not DSL enough. Let's extract the condition to a one-liner method, and make the method name suitable to be appeared after if.

    if (feeIsBasedOnPercentage(feeType)) {
        ...
    }

Much better, right? Now the question is, if the condition needs further explanation, why not explain it directly with a comment side by side, like this.

    if (feeType == FeeType.PERCENTAGE) {
        // fee is based on percentage
        ...
    }

No, this's not good. The comment looks like redundant. It just repeats what's in the if statement. Okay, let's remove the redundant comment.

    if (feeType == FeeType.PERCENTAGE) {
        ...
    }

So, what was the question again?

If you find yourself extract / write

• a very short, usually one-liner, method, and
• it's only called once, and
• method name is repeating what's in the body

you're actually writing a redundant comment, but at a different place from what it comments on, like this.

    private boolean feeIsBasedOnPercentage(FeeType feeType) {
        return feeType == FeeType.PERCENTAGE;
   }

Finally, I'd like to guess how this redundant way of programming was developed, just for fun.

   private boolean feeIsBasedOnPercentage(FT ft) {
       return ft == FT.P;
   }

Hope you get the idea of why I call it redundant. If you do, Dependency is the new Inheritance is another one for you.

Disclaimer, code snippets used here are for illustration only.

Are you setting Connect Timeout correctly?

Application development nowadays is becoming more and more like mashing up. Unless you provide file storage service, or data repository service, there's no way you can avoid consuming 3rd party APIs / (micro-) services. In Java battlefield, most of the time, internal or external APIs are provided via RESTful interface. If this is the case, chances are to consume a web service you'll be using one of

• org.springframework.web.client.RestTemplate
• org.apache.http.impl.client.HttpClientBuilder, or its cached version
• org.apache.http.impl.client.cache.CachingHttpClientBuilder
• org.apache.http.impl.client.DefaultHttpClient, or its fluent version
• org.apache.http.client.fluent.Request

No matter which one you end up use, it's always a good practice to set timeout for HTTP clients, and this can be done only by org.apache.http.client.config.RequestConfig. Now can you please check the value you pass into RequestConfig.Builder#setConnectTimeout()? If it's significantly larger than 5(ms, not a typo here), you're not setting it correctly.

Connect timeout is used to provide QoS of the creation of TCP connection between client and server, not the whole lifecycle of the connection, or in pooled environment, from borrow to return. If this process can't finish in few milliseconds, org.apache.http.conn.ConnectTimeoutException should be thrown to speed up the exception handling. It's not the end of the world however, if you misuse it. It's just better to fail fast in case something goes wrong for better user experience, rather than wasting the x seconds you set.

Then how about the RequestConfig.Builder#setSocketTimeout()? Well, I may have another post on it. Until then, it's more like the meaning you think setConnectionTimeout has.

Never take anything for granted.

Glad I'm of some help

I'm no way near a productive blogger. Even though I've got only 80 posts in nearly 5 years, some of them are referred by more popular bloggers, which is definitely helping more people and, at the same time, giving me the credit.

Empathy and Samba by http://www.abclinuxu.cz/poradna/linux/show/287722

Schema of iTunes podcast feed by http://wiki.bcmoney-mobiletv.com/index.php?title=ITunes

Let scim and ibus support Skype by http://zazchor.me/blog/2013/09/13/fix-ibus-not-working-with-skype-4-dot-2-in-ubuntu-12-dot-04/

How to install Adobe Flash plugin on Ubuntu 12.04 by http://howaryoo.wordpress.com/2012/08/10/ubuntu-12-04-lts-new-install-oracle-java-and-flash/

Java development using Ubuntu 11.10 and OpenJDK 7 by https://twitter.com/taq/status/125972118000242689

Let's keep up sharing / open-sourcing knowledge.

Maven 3.1 and 3.2 are incompatible with 3.0

As title says, Apache Maven 3.1.x and 3.2.x are not compatible with 3.0.x, and this may affect your artifacts.

The main difference is whether or not, the default (compile) scope dependencies in a test, or provided scope dependency will be included in the final artifact, if you know what I mean.

In Maven 3.0, all the compile scope dependencies in test or provided scope dependencies will be included, unless they're overwritten by others in any ways.

In Maven 3.1 and 3.2 however, all the compile scope dependencies in test or provided scope dependencies will NOT be included, unless they overwrite others in any ways.

Personally, I like what Maven 3.1 and 3.2 are doing. Just a heads up you may get hurt by including ugly designed test or provided dependencies.

Dependency is the new Inheritance

To warm up, please have a read how bad inheritance is.
disadvantages of inheritance in java
Why extends is evil

Okay, dependency is the new inheritance, just even harder to do it right.

With inheritance, you extend something means you decide to be it, just a bit special. It's more base classes' responsibilities to make sure it doesn't become something else. With dependency however, you introduce a dependency just because it has something that's useful to you, no matter how much in it you don't need at all. Check the exclusions in pom.xml to see how many dependencies, direct or indirect, you included and later you found they caused trouble. Isn't this the new Refused Bequest, except there's no better way to solve it?

With versioned dependency, you changed something but others depend on it don't know if it breaks their part. "If it ain't broke, don't fix it". They choose to stick with the old version of what you've changed, and not to move forward with you. Without their following and feedback, you make further changes more freely, knowing that only part, if any, of your users bother to follow what you're doing.

Popular languages, like C# and Java, only allow single inheritance to avoid the potential problem caused by multiple inheritance. This makes inheritance less dangerous for everyday use. But you are free to depend on no matter how many dependencies. Very likely, those dependencies depend on difference versions of same module, directly or indirectly. Then it becomes your job if this doesn't work and you have to manually exclude all but one, the one that can fit all. Put it simply, you have to manage and solve dependency conflict, which is never part of your job in single inheritance paradigm.

Hope I'm clear and to the point by now why dependency is the new inheritance. Versioned dependency is the root of Dependency Hell? Not really, but you must have the skills and experience to handle it.

multiple XML APIs in project

If you have below exceptions in tests or runtime, it's cause by multiple XML APIs / implementations in your project. How to find and exclude them is beyond this post.


ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[resteasy-servlet]] (http-0.0.0.0-8080-15) Servlet.service() for servlet resteasy-servlet threw exception
java.lang.LinkageError: loader constraint violation: when resolving field "DATETIME" the class loader (instance of org/jboss/classloader/spi/base/BaseClassLoader) of the referring class, javax/xml/datatype/DatatypeConstants, and the class loader (instance of ) for the field's resolved type, javax/xml/namespace/QName, have different Class objects for that type
at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.(RuntimeBuiltinLeafInfoImpl.java:270)


java.lang.IllegalStateException: Failed to load ApplicationContext
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.checkOverrideProperties(ClassBeanInfoImpl.java:205)
at com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.(ClassBeanInfoImpl.java:186)
at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:509)
at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:528)
at com.sun.xml.bind.v2.runtime.property.ArrayReferenceNodeProperty.(ArrayReferenceNodeProperty.java:87)

You're What You Depend On

People love 3rd party libraries, especially those from Apache, Google, etc. But I'd like to ask a question, how do you really find vanilla Java SDK hard to use?

The reason I ask the question is, I found a dependency of com.google.guava:guava:13.0.1 in a project and the only usage of Google Guava throughout the whole project is:
import static com.google.common.base.Charsets.UTF_8;

Wow! I'm sure the programmer thought the same when he decided to make the project depend on Guava. I'm also sure he was so excited and didn't even bother to check the magic implementation in Guava:
import java.nio.charset.Charset;
public static final Charset UTF_8 = Charset.forName("UTF-8");

I feel sorry for the programmer, not only because he introduced an unnecessary dependency, but also he lost a chance to learn from 3rd parties to master Java SDK better.

MongoDB-based Cache Service

I've talked about using REST web service to wrap database and provide managed repository service. This time, I'd like to discuss developing cache service with MongoDB's two convenient features.

Before I start, I want to make it clear that the cache service here is not the one that trying reduce response time to sub-ms. It's something you hesitate to call it again and want to store it somewhere. Usually it's a distributed web service (for example, Maps API from Google), or an expensive SQL statement result. You want to cache it not only because you don't want to wait for few seconds again, but also to save your usage quota, or reduce the workload of a database. In this case, you'll be happy if we can reduce the response from X seconds to X ms.

Depending on whether it's a single node or clustered environment, the size of the cached data, text or binary, there are quite a few products that can fulfill the task. But when we examine if the solution can scale up and scale out, the answer become not clear. Consider config something like Memcached in a 4 node cluster and you'll get the idea. Basically you have to explicitly tell each node, "you are in a group so you guys have a shared memory or disk".

How about share nothing? As long as a node knows the cache, it doesn't matter how many other nodes also know the cache, they can share something, with same key of course. Cache service can just become a couple of HTTP methods (POST and GET) backed by MongoDB. But why MongoDB?

One aspect of a cache is the capacity, in bytes or in number of objects. In MongoDB, you can use Capped Collections to achieve this. You can create a capped collection using
db.createCollection("mycoll", {capped:true, size:100000})
or convert a collection to capped one using
db.runCommand({"convertToCapped": "mycoll", size: 100000});

The value of size parameter is in bytes. You may not know the size of a document in the collection if you want to know how many documents can be stored in the capped collection. If you already have a amount of documents, you can run
db.mycoll.stats()
and check the avgObjSize value before converting it to capped collection. Here is an example,
{
    "ns" : "mydb.mycoll",
    "count" : 7739,
    "size" : 42885120,
    "avgObjSize" : 5541.429125209976,
    "storageSize" : 65724416,
    "numExtents" : 8,
    "nindexes" : 1,
    "lastExtentSize" : 23224320,
    "paddingFactor" : 1,
    "systemFlags" : 1,
    "userFlags" : 0,
    "totalIndexSize" : 228928,
    "indexSizes" : {
        "_id_" : 228928
    },
    "ok" : 1
}

If you run stats() on a capped collection, you'll see 2 more lines in result

    "capped" : true,

    "max" : NumberLong("9223372036854775807"),

Another feature in caching is Time To Live, which is used to specify when a cached item should be invalidated. In MongoDB, you can create index on a date field and provide expireAfterSeconds option to set the TTL of a collection.
db.mycoll.ensureIndex( { "created": 1 }, { expireAfterSeconds: 3600 } )

Note however that the background task to delete expired documents runs once every 60 seconds, so don't expect this feature working much more accurately than that. And you can't make a collection both size- and time-based (who's going to need both anyway).

So next time when you design a size-based or time-based cache, would you like to consider MongoDB?

Dependency Hell

Like Dependency Lock-in, another big problem in modular software development is Dependency Hell. I have a very recent example.

 +- org.apache.httpcomponents:httpclient:jar:4.2.3:compile  
 | +- org.apache.httpcomponents:httpcore:jar:4.2.2:compile  

You may be familiar with both and ask why I don't have the same version of httpclient and httpcore. Because that's the ways they work together.

If you use httpclient and also declare httpcore in your pom.xml, congratulations, you're making Dependency Hell. But what if you don't declare httpcore but httpclient gets updated and removes the dependency to httpcore, or depends on something else that provides http core function? Good luck. Spring Framework 3.2.1 made such a mistake and broke one of my hobby projects. The hell has different impact on component provider and consumer, but most of time, we're both component provider and consumer.

You shouldn't care about a component that your dependency depends on, and you shouldn't let any user of your component care about any component that your component depends on. If you break this, you break the fundamental principle of software component design. But in practice, unless you use OSGi [update 18/6/2020 - or Java 9 Platform Module System (JPMS)], you don't have easy ways to control how not to expose your dependencies to those components that directly, or indirectly, depend on your component.

Versioned jar files are the root of Dependency Hell in Java development. Check your pom file to see how many dependencies are redundant, they should be taken care of by direct dependencies; And how many dependencies should be upgraded to latest but you can't. Consider replace these outdated jar files with XaaS services?

What an average GWT team looks like?

From a survey result, an average GWT team looks like this. If your team don't deliver and you're thinking try something else, compare this with your team.

What do you think? Let me guess. "My mileage varies", right? :-)

Different approach to prevent SQL Injection and XSS

From security point of view, a system should prevent SQL injection, cross-site scripting and other potential vulnerabilities. But from a developer / architect point of view, the approaches to prevent them are quite different.

One might say that as long as we can stop any SQL injection and XSS strings from entering into a system (exhaustive prevention method?), we win. I'd prefer defer the prevention until as late as possible, just before the code is about to execute. That said, the architect of a system shouldn't try to list all the ways information goes into a system, and predict what could be the new ways in the future.

For SQL injection prevention, the only check point should be in DAO layer. For XSS prevention, the only check point should be in Web presentation layer. These are the ultimate solutions.

Yes, you can check every user input for SQL injection, but what about all the inbound messages and documents from messaging system, all the responses from 3rd party web services? As long as the information won't go into database, drop table is not a problem at all. Likewise what's the point to prevent script alert('attacked') /script from being printed on a printer?

On the contrary, you check all the input for XSS, but by accident you DBA (who knows every detail about SQL injection) inserts script alert('attacked') /script into database and this field will be presented in browsers. Can your system handle this?

That being said, dump the exhaustive prevention method. drop table is not a threaten as long as it never hits your database; script alert('attacked') /script isn't either as long as it's never presented in a script runtime.

Software imitates nature

In some software I participated, a web application is actually divided into two projects during development, and two deployable artifacts during runtime. This design addresses the idea that the UI of an application should only be the presentation of the application, while all the logic keeps the same in back-end even when front-end UI is changed. In this case, a lot of communication between front-end and back-end are required. And a lot of problems are also caused by this kind of communication. Let me give you an example.

There is a list of items in browser. User can make changes to the list, add items to the list or remove items from the list. Each time the list is changed, an asynchronized RESTful request is sent to back-end to update the model. Everything seems fine but sometimes it works strangely as the front-end and back-end have different items of the same list.

We finally figured it out that this is because the items in the list might change before the previous request is finished, and the change may or may not be sent to back-end and processed in order. Solving the problem is easy once we know the reason. Developers decided to temporarily disable the list from being modified until previous request is finished. Problem solved, well, for this specific front-end.

But if we turn to real life for such scenario, helmsman repeating verbal commands is one example to avoid misunderstanding between two parties. Back to the problem, to keep single source of truth, we can also return items in list as response and request any front-end to update the items in list to be the same.

You might wonder why bother doing the same thing twice. It's because they have different point of view. Disabling changes in front-end while waiting is a solution for a front-end not to have different state from back-end. Replying the state in back-end is telling any front-end what's in back-end is the only truth. Following the latter one is the key, while the former one is just a user-friendly add-on.

This is just an example that many software problems are really projections of real life problems that already got solved. Don't try to re-solve them when you can simply follow patterns that are already proved in nature.