This blog post is about URL shortners and specifically my URL shortner (MyURL) based on the infrastructure of AmmarServer
For more information on AmmarServer you can read this

URL shortners are internet tools that take long and impossible to remember URL strings like http://www.youtube.com/watch?v=lR9mNuLIPBU and add an easier to remember (or a random) alias that makes it shorter..! like http://tinyurl.com/d69n5sr or http://myurl.ammar.gr/fadetoblack ..

This web-service of course is a pretty straightforward one since it is just a lookup table with aliases pointing to strings and thats about it ,on the server side.. Thats also the reason why it is one of the first “service applications” of AmmarServer..

The code itself is simple and at the time around 420LOC , though I would like to reduce them even more..

The whole server executable is around 90KB ( taking efficiency seriously ;P )
and apart from the server binary there is also a plain-text file called myurl.db which contains the aliases and real links. Literally these are the two things you need and you got yourself a URL shortner.
Of course the db file is read only one time on startup and it is written every time a new alias is added from a user.

Basically what we do is spawn an AmmarServer instance on port 8080 and use an apache proxy to redirect traffic going to myurl.ammar.gr to 127.0.0.1:8080 . This can be easily done by enabling mod_proxy ( sudo a2enable mod_proxy , sudo a2enable mod_proxy_http ) and by adding a virtual host like the next code segment

<VirtualHost *:80>
ServerName myurl.ammar.gr
<Proxy *>
Order deny,allow
Allow from all
</Proxy>
ProxyPass / http://127.0.0.1:8080/
ProxyPassReverse / http://127.0.0.1:8080/
</VirtualHost>

( Note : I would really like to make a proxy based on AmmarServer and probably will do to be able to get rid of apache completely )

Of course as I do with most things I tested MyURL against other popular URL Shortners , ( namely tinyurl.com , bit.ly , goo.gl , tr.im and ow.ly ) and found out a good performance advantage ..

As demonstrated below :

Task Shortening ->

time wget http://www.youtube.com/watch?v=lR9mNuLIPBU
2013-04-19 17:05:03 (840 KB/s) – `watch?v=lR9mNuLIPBU.1′ saved [232418]
real 0m0.755s
user 0m0.000s
sys 0m0.008s

time wget http://myurl.ammar.gr/fadetoblack
2013-04-19 17:50:19 (12.7 MB/s) – `fadetoblack.7′ saved [127/127]
real 0m0.066s
user 0m0.000s
sys 0m0.000s

time wget http://tinyurl.com/d69n5sr
2013-04-19 16:15:08 (815 KB/s) – `d69n5sr’ saved [232370]
real 0m1.082s
user 0m0.000s
sys 0m0.008s

time wget http://bit.ly/plIyZu
2013-04-19 16:18:03 (716 KB/s) – `plIyZu’ saved [232383]
real 0m1.038s
user 0m0.000s
sys 0m0.008s

time wget http://goo.gl/XAQqU
2013-04-19 16:19:22 (742 KB/s) – `XAQqU’ saved [232387]
real 0m0.825s
user 0m0.004s
sys 0m0.008s

time wget http://tr.im/42d2q
2013-04-19 16:23:29 (825 KB/s) – `42d2q’ saved [232693]
real 0m1.218s
user 0m0.000s
sys 0m0.012s

time wget http://ow.ly/kdVRX
2013-04-19 16:36:15 (806 KB/s) – `kdVRX’ saved [232907]
real 0m1.209s
user 0m0.004s
sys 0m0.008s

Of course this is not an accurate benchmark since myURL currently only has around 30 aliases while the other services have possibly millions , and also I am testing this from Crete , Greece , and the myURL server is in Athens Greece , while most of the others are on the US ( which implies a lot of “jet”lag ) , and network congestion might vary from time to time..

Then I took a look on the filesizes of the responses , which to my amazement was ~300KB .. Turns out that due to the Redirection using 301 headers wget was actually going on and retrieving the full youtube page so i then went on subtracting this time from each of the responses which yielded

myURL 0.066s <- myURL performance
tinyurl 0.327s
bit.ly 0.283s
goo.gl 0.07s <- this is also awesome
tr.im 0.463
ow.ly 0.454s

So that being said , considering my non-special hardware server , my plain DSL connection , my DynDNS DNS , my D-Link ADSL modem doing NAT and serving the data , myURL performance is overall superfast !

One question is how the f*$# is google going “as” fast ( and while providing a better service , analytics , user based “tracking” ;P etc ) and another one is how much faster could I go !
When running a test through a local loopback connection I get

time wget http://127.0.0.1:8080/fadetoblack
2013-04-19 17:27:05 (5.55 MB/s) – `fadetoblack.1′ saved [127/127]
real 0m0.010s

which is the maximum throughput the server can go at ( it is 7 times less than google’s net time ) If I only knew a little bit about how fast goo.gl go from inside their LAN , or what their software architecture is like

You can find MyURL here -> http://myurl.ammar.gr/ , although it is in alpha stage and thus will probably not be always online ( as I am performing tests on it , recompiling ,restarting etc ) ..!
Also ammar.gr is not exactly a high-availiability site , mainly thanks to my ISP
Of course as mentioned above myurl.ammar.gr is the same as calling http://ammar.gr:8080 . Of course calling myurl.ammar.gr also adds an overhead for the proxy thing to take place but for now it will have todo , Also I might change ports in the future so myurl.ammar.gr will be the thing to remember..

What I wanted to record somewhere for someone that might face the same problem is the very strange behaviour of OpenNI after I installed ROS and its OpenNI libraries.. ( sudo apt-get install ros-groovy-openni-camera ros-groovy-openni-launch )

I’ve been getting some really weird behaviour , namely an Error (” Bad Parameter sent to the device! ” ) after the
StartGenerating call on the image generator.. This happens everywher including standard binaries of the PrimeSense codebase like NiViewer , but as well as my own wrapper ( https://github.com/AmmarkoV/RGBDAcquisition )

After some extensive tinkering it turns out that after ROS gets installed it takes over the “OpenNI stuff” and in order to use it as usual you have to first initialize it the ros way.. The weird thing is that this does not impact getting Depth frames that work correctly..

If you want to use it you should first initiate :

roscore
roslaunch openni_launch openni.launch &

and test if it works using -> rosrun rqt_reconfigure rqt_reconfigure

The same library with no modifications works correctly after the steps mentioned

I have also found some similar problems that might be different manifestations of the same problem..
So I really hope this helps someone, it took me almost two days to find the solution ..!

http://answers.ros.org/question/54907/openni-error-opening-video-stream/

The idea is a tool for console based content-aware image searching much like the experience when using google advanced image search.. All this from the comfort and privacy of your terminal screen..
Of course such a cli tool could be leveraged by a GUI application to provide additional searching services in a more “user-friendly” way , and with that beeing said I am planning to make use of it in my FlashySlideshows project ..!

At last my huge random 4chan image folder will become automatically arranged!
I litteraly don’t know what I will do with all the time this will free ;P

sample queries using the library/tool are :

Gathering all the images that “look like” 1360273766703.jpg ( a meme from meme generator ) into the Downloads/MemeTest dir..!

mv `lsimg -likeExt ~/Downloads/1360273766703.jpg 30 60 ~/Downloads/` ~/Downloads/MemeTest/

Querying all “red” images
by using
lsimg -color red ~/Downloads/

Querying all images big enough for being used as a wallpaper
by using
lsimg -minDims 1920 1080 ~/Downloads/

Of course you can combine filters together for more complex queries and there are more parameters coming up ( like face detection , containers , GPS EXIF info parsing etc )

The project is licensed under GPLv3 , it is only targeted for the Linux platform
and you can download it from here : https://github.com/AmmarkoV/AdvancedImageSearch

(or by issuing)

git clone https://github.com/AmmarkoV/AdvancedImageSearch.git
cd AdvancedImageSearch
./get-dependencies.sh
./install.sh

The project currently only works on jpeg images (since it is in early alpha stage) and there are no other dependecies except libjpeg for now
I will probably have to add openCV linkage for use of the composite openCV face detectors and misc other stuff I want to include..

That’s it for now..!
Hope you enjoy..!

Actual Sony patented technology

Playstation 4 was announced today after a show that was broadcasted live over the internet..
Unfortunately I can’t seem to find a recording of it somewhere , but when I do I will append it here..

Although I grew up playing with PS1 played only a little with PS2 and owned a PS3 it is now apparent that I will not be getting the PS4 ..

The main things that I didn’t like about it are..

- The controller “share” button! ( I wonder who came up with that idea.. )
- The fact that you can’t “share” your games with your friends any more (literally share by lending them the disc)
It appears this will not be the case , or so I read from news sites on the internet
- The social media , free advertisment on your friends , on you , on everyone attitude
- The “it will be solved in the cloud” approach on backwards compatibility with older generation devices ( let alone games you already purchased )
- The fact that you can’t turn your own device off unless you pull the plug.
- The mandatory social media logins , with everything this implies..
- The secondary “custom-chip” that performs “background-processing” , linked to your social media :p
- The fact that it will download things that it will decide its user might want in the future ( I bet it could also torrent-like share them with other users all in the name of “better performance” and at the users bandwidth cost )
- The playstation eye having a proprietary auxilary connection to the console and not a USB one..
- There were no price reports or “hands on” view..
- The marketing glossary of the presentation..

What I do like is :
- The video encoding “co-processor” for remote-operation ( although the PSVita sucks and it is a Wii-U ripoff idea )
I believe “hardware accelerated graphics output forwarding” would be a great platform-agnostic addon to modern digital devices , along with a way to transmit control messages it could definately be useful. Of course ssh and X-Forwarding are great but if you want to stream 3D graphics and full-hd movies they are kind of problematic..
- The stereo approach on 3d viewing , it will be nice in a future with devices like oculus rift or google glasses , if PS4 eye is of the same quality as PS3 eye it will be a great device , properly calibrated and synchronized , I really hope it will be usable by non PS4 hardware..
- The developers will probably have an easier job optimizing their code for all their platforms ( since there is basically only one common platform architecture ) , although if the playstation SDK and the black box it comes with costs nearly as much as the PS3 one ( 7500Euro+ ) , I still don’t think PS4 games will become easier to develop

What I’d like to note is that everything from Mac’s to Consoles is a PC nowadays..
As I said being a PS3 owner , I will not buy a PS4. I would like to buy its camera system if there was a way to connect it with a normal pc

That being said I am really looking forward to the Microsoft answer ( X-Box 720 probably streaming to their Surface tablets or w/e ) and the Steambox or “Piston” which is the platform that hopefully will win the 8th generation console wars..

AVX instructions support 256bit payloads and that is fantastic , the newest versions of gcc automatically utilize the new instructions ( when gcc is configured to produce output for this architecture ) and senses loops that can be “vectorized” , what is very handy is that you can monitor this kind of optimizations using -ftree-vectorizer-verbose=1 which prints out messages that indicate where the optimizations take place ( providing a high level optimization overview )

I have created a new repo on my github account so that anyone who might want to test these kind of things will find a codebase that might make it easier..

One thing that I also found extremely useful is the Intel Intrinsics Guide which is a very helpful guide for looking up CPU OPcodes , and intrinsics with very simple explanations and plain code equivalents..!

My first test was a frame subtraction scenario repeated 1000 times for two FullHD frames..
Its code is the following ..

#include
#include
#include

int SubtractImages(char * imageA,char *imageB,unsigned int elements)
{
char * imageA_Ptr = imageA;
char * imageA_Limit = imageA + elements;
char * imageB_Ptr = imageB;
while (imageA_Ptr {
if (*imageA_Ptr>*imageB_Ptr)
{
*imageA_Ptr-=*imageB_Ptr;
}
else
{
*imageA_Ptr=*imageB_Ptr-*imageA_Ptr;
}

++imageA_Ptr;
++imageB_Ptr;
}
return 1;
}

int main()
{
unsigned int width = 1920;
unsigned int height = 1080;
unsigned int frameSize = width*height*3;

char * imageA = (char * ) malloc(frameSize);
char * imageB = (char * ) malloc(frameSize);

unsigned int benchTimes=0;
for (benchTimes=0; benchTimes<1000; benchTimes++)
{
memset(imageA,123,frameSize);
memset(imageB,123,frameSize);

SubtractImages(imageA,imageB,frameSize);
}

return 0;
}

Running ./make.sh ( see here )
produced the following two assemblies and binaries for the same “job” , one plain and unoptimized and one automatically optimized for core i7 AVX CPUs

Running the two binaries ( using ./test.sh ) the results where

Running AVX binary
Command being timed: “./SubtractionAVX”
User time (seconds): 0.76
System time (seconds): 0.00
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 12588
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 3189
Voluntary context switches: 1
Involuntary context switches: 67
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

AND

Running Normal binary
Command being timed: “./Subtraction”
User time (seconds): 22.74
System time (seconds): 0.00
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:22.79
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 12592
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 3191
Voluntary context switches: 1
Involuntary context switches: 1920
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

A 30x speed up! The SIMD architecture is a great tool indeed!
After this first run I wanted to try and write my own hand “optimized” calls and see if I could go better than the gcc compiler , but after taking a look on the optimized assembly it appears that I wouldn’t be able to do better , so I will have to come with a more complex original code segment that will not be so efficiently auto-optimized and then try my luck there..

An example of operations that would be more difficult to auto-optimize with gcc is this block matching comparison using SSE4 and comparisons for using up to 16×16 windows

Another great article on the Intel site is this “vectorizing for struct” advice

To go back to my main topic though , here are the two assemblies :
Plain Assembly , without SIMD optimizations
AVX , highly optimized assembly ..!

The difference of course is made by this segment which utilizes the better instruction set on the second binary

…
vmovdqu (%rdi,%rcx), %xmm1
addq $1, %r9
vmovdqu (%rsi,%rcx), %xmm0
vpsubb %xmm0, %xmm1, %xmm3
vpsubb %xmm1, %xmm0, %xmm2
vpcmpgtb %xmm0, %xmm1, %xmm0
vpblendvb %xmm0, %xmm3, %xmm2, %xmm0
vmovdqu %xmm0, (%rdi,%rcx)
addq $16, %rcx
cmpq %r9, %rax
ja .L4
leaq (%rdi,%r10), %rax
cmpq %r10, %r11
leaq (%rsi,%r10), %r8
je .L11
…

Using the Intel Intrinsics Guide ( mentioned again above )
we see that

Instruction: vmovdqu
CPUID Feature Flag: AVX
Description
Load 256-bits of integer data from memory into dst. mem_addr does not need to be aligned on any particular boundary.
Operation
dst[255:0] := MEM[mem_addr+255:mem_addr]
dst[MAX:256] := 0

Instruction: vpsubb
CPUID Feature Flag: AVX2
Description
Subtract packed 8-bit integers in b from packed 8-bit integers in a, and store the results in dst.
Operation
FOR j := 0 to 31
i := j*8
dst[i+7:i] := a[i+7:i] – b[i+7:i]
ENDFOR
dst[MAX:256] := 0

Instruction: vpcmpgtb
CPUID Feature Flag: AVX2
Description
Compare packed 8-bit integers in a and b for greater-than, and store the results in dst.
Operation
FOR j := 0 to 31
i := j*8
dst[i+7:i] := ( a[i+7:i] > b[i+7:i] ) ? 0xFF : 0
ENDFOR
dst[MAX:256] := 0

Instruction: vpblendvb
CPUID Feature Flag: AVX2
Description
Blend packed 8-bit integers from a and b using mask, and store the results in dst.
Operation
FOR j := 0 to 31
i := j*8
IF mask[i+7]
dst[i+7:i] := b[i+7:i]
ELSE
dst[i+7:i] := a[i+7:i]
FI
ENDFOR
dst[MAX:256] := 0

GCC did a great job indeed

Steam for linux launched today with some very low prices on a “games special offer” for the next week..

Although I already have many of the games that are featured on steam ( and I have them DRM free from humble bundle ) , counter strike source priced at 3.6 euro was really a bargain so I purchased it and played a little like old times..

Unfortunately NVIDIA optimus users will have issues with steam as its model of launching games doesnt take into account optirun , etc..

One quick fix that makes everything work out of the box for optirun users , but does so using the Intel card is the following

ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 ~/.steam/steam/ubuntu12_32/

The intel card is more than enough for Counter Strike Source

I also tried to launch the game as stated below ( Using bumblebee and the NVIDIA card ) but it would start in LAN only mode , and since I don’t have someone to play localy it there wasn’t much point in it..

cd “~/.steam/steam/SteamApps/ammarkov/Counter-Strike Source/” &&
export LD_LIBRARY_PATH=~/.steam/steam/ubuntu12_32:”~/.steam/steam/SteamApps/ammarkov/Counter-Strike Source/bin/” &&
optirun ./hl2_linux -game cstrike -console -dev -numericping -noipx

A great documentary about money , banks , loans , securities and a retrospect for the financial situation today

See it @ Youtube
IMDB
Wikipedia

The Pirate Bay AFK , Torrent
The Pirate Bay AFK @ Youtube

The good news is that you can work around this by installing an older version of the gcc compiler , and the even better news is that you can do everything with two small scripts I wrote that handle all the fuss for you and that you can download below..
Please note that the scripts will automatically check your machine capabilities ( x64 x86 ) , download the SDK you want from NVIDIA if you dont have it , and even set-up the correct LD_LIBRARY_PATH and PATH strings on your .bashrc file..
Come on , what more can you ask for ?

I strongly suggest not downloading new drivers when the CUDA installation will ask you for it but use your distribution packaged ones (nvidia-current).. The repository drivers on Ubuntu 12.10 are fully equiped for CUDA so there is no reason to change them . You should also be aware that if you are running on an Optimus Laptop GPU ( see an older post here on configuring bumblebee for optimus cards ) installing standalone new drivers will most likely break your graphics setup.. So once again this time specifically you Optimus users dont download new drivers when the cuda installation asks you because you will break graphics on your Ubuntu installation
Also in case you are curious Optimus systems ( just like mine ) will run the script with no problems just as the rest of the NVIDIA systems .. The only difference on Optimus Chips is that after you compile an executable with nvcc you will have to run it using optirun ./yourexecutable ..

CUDA 5 ( if you want to take a look on the script click here )
To install CUDA5 type the following commands on your terminal :
wget https://raw.github.com/AmmarkoV/MyScripts/master/Setup/install_cuda5
chmod +x install_cuda5
./install_cuda5

CUDA 4.2 ( if you want to take a look on the script click here )
To install CUDA4.2 type the following commands on your terminal :
wget https://raw.github.com/AmmarkoV/MyScripts/master/Setup/install_cuda4.2
chmod +x install_cuda4.2
./install_cuda4.2