Saturday, September 19, 2015

Adding new machine to rc files sync

Some time ago I wrote a post on linux productivity tools I use.
As from time to time I need to install new machine that would benefit from the synced rc files, I’ll document the steps required (as rediscovering it every time is rather a waste of time).

What I’d need

I’ll use the previously introduced:
* oh-my-zsh
* homeshick
* vim’s vundle plugin and
* tmux with tmuxinator
* powerline fonts

Please note: order is important here, otherwise some install steps might fail.

oh-my-zsh

Install via:

sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

homeshick

Install via:

git clone git://github.com/andsens/homeshick.git $HOME/.homesick/repos/homeshick

Get existing dotfiles via:

cd $HOME/.homesick/repos
$HOME/.homesick/repos/homeshick/bin/homeshick clone <your git repo>

For me, as I use the private bitbucket repo, I just had to add the ssh keys generated to bitbucket first:

  1. generate ssh key via (following https://help.github.com/articles/generating-ssh-keys/)
  2. go to repo on bitbucket via web browser
  3. add generated key at: Settings -> Deployment keys

And run the

cd $HOME/.homesick/repos
$HOME/.homesick/repos/homeshick/bin/homeshick clone git@bitbucket.org:typekpb/dotfiles.git

Make sure to let homeshick symlink all the dotfiles from the repo (you might want to back up those to be overwritten). You’ll be prompted for symlinking stuff.

vim’s vundle plugin

As a prerequisite you’d need (g)vim installed (install instructions would be distro specific).

Install vundle plugin via:

git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim

Afterwards install all the vundle managed plugins via running in the vim:

:BundleInstall

tmuxinator

Make sure to install tmux and ruby first (probably from the distro packages available).

gem install tmuxinator

if it’s to be installed globally, use:

sudo gem install tmuxinator

powerline fonts

Following the official instructions:

cd /tmp
wget https://github.com/powerline/powerline/raw/develop/font/PowerlineSymbols.otf
wget https://github.com/powerline/powerline/raw/develop/font/10-powerline-symbols.conf
mkdir ~/.fonts
mv PowerlineSymbols.otf ~/.fonts/
fc-cache -vf ~/.fonts/
mkdir -p ~/.config/fontconfig/conf.d/
mv 10-powerline-symbols.conf ~/.config/fontconfig/conf.d/

after restart seems to work OK.

Done :)

Tuesday, January 6, 2015

Ubuntu 14.04 with HP Deskjet 930c

I’ve got old printer: HP Deskjet 930c. Following steps were necessary to use it with my Ubuntu 14.04:

  1. Printer: start + connect it via USB
  2. download HP drivers bundle called: HP Linux Imaging and Printing from: http://hplipopensource.com/hplip-web/gethplip.html
  3. install bundle via (answering couple yes/overwrite questions):
chmod +x hplip-3.14.10.run && ./hplip-3.14.10.run 

You should see your new printer afterwards.

Friday, October 24, 2014

Linux productivity tools for me

This is the stack/are the tools that keep me productive in my day to day programming experience on linux.

Well, first of all let me tell you, that this might not fit your needs , as I’m a full time java programmer so have OS choices open (still I won’t go to my java-related stuff here, to make infos general and possibly usable for non-java guys as well.).

Linux is my choice for quite some years already and this post won’t detail the reasons, rather will focus on tools/utilities that help me survive in this world.

Distro

Well, let’s start from the ground. I don’t have a clear winner here. In fact I run currently 3 different ones (didn’t count my router and phone here):

In fact the info present in this post won’t detail too much on the 3.rd one, as it’s a server that just runs. The only thing I need there is a ssh access and that’s pretty much it, so this post would be rather short for that one :)

The reasons for Fedora, and Ubuntu in my case are:

  • stability (well, let’s say things don’t crash that often for me),
  • packages for most of the SW I need available in repositories and
  • the software packages are not at some archaic versions.

Window manager

Well, the nice thing is that most common window managers are available across all the popular distros. My choice here is Xfce.

Well, I don’t hate others, this just fits my needs best:

  • no need for fancy effects for me as
  • speed is rather my preference.
  • still I need something that I would not fight with every day.

Project I’m keeping my eye on is: LXQt. Still waiting for a more stabilized release (0.8 was released recently, which says to be ready for production desktops, but I’d plan to give it a try with the next release as well as when I have some time to set it up).

Shell

I have 2 candidates here:

  • bash and
  • zsh.

bash I use for scripting. It’s been my choice for quite some time already and I don’t see a need for change any time soon. As I’m quite used to it and the most people/projects I share the scripts with are OK with that/have used it already.

zsh as my default shell. This is still the world I consider new to me (see my recent post on that).

Shell env sync

As I have multiple machines I’m working on, there are 2 essential projects in the area for me:

oh-my-zsh

Provides for my zsh:

  • nicely/systematically structured plugin approach,
  • autocompletition and
  • all sorts of aliases.

To document my use case a bit here, these are the plugins I have present in my .zshrc:

plugins=(git mvn glassfish yum colored-man vagrant z common-aliases gradle homeshick vim-interaction powerline tmuxinator tmux)

Please note: Some are not available officially, and in the time of writing are just present as pull request from me to the project. Namely:

Well, might sound like a self promotion, but feel free to check and provide feedback on these if you find them useful.

Homeshick

Lets me synchronize all my custom zsh plugins as well as .bashrc files over git repository.

The only requirements are:

  • git and bash on all the clients
  • and the git repo accessible from all my clients.

In my case, there are things I don’t want to expose to public, but have no problem to have it hosted in some private git repo. So bitbucket is my choice, as I have there private git repo for free for the purpose.

Shell sessions bootstrapping

There are 2 important projects for me here:

Tmux and tmuxinator enable me to have just one yaml formatted file for bootstrapping my terminals.

I preffer it to having multiple tabs open in some GUI terminal, as it:

  • bootstraps all my shells daily in a same/reproducible way,
  • provides me with the nice way on organizing these and
  • navigation between these works with keyboard only => switching between different tasks becomes after some time of usage just a routine.

So no more searching in countless cards/windows for particular task.

Shell productivity

I don’t want to list all the linux utils I use, as I guess I’d make list too long and too boring (even for me to write). So let’s just name some, that I consider worth it:

  • ack - powerful grep replacement (I might document in the separate blog post),
  • notify-send and
  • all sorts of aliases and zsh/bash functions, that would be too many to list here, moreover might be too specific and useless for others.

Notify-send

I blogged about notify-send already.

I use it for all the app-server (in my case Glassfish) lifecycle management operations (start/stop/restart domain) as well as deployment stuff.

As these:

  • take quite some time to finish and
  • might end up with errors,

I let it myself updated with notification having:

  • exit code
  • chunk of the last couple lines from the log file.

This is great, as I can work in parallel and it gets my attention, once the job is done.

Desktop app launcher

Without having an option for fast startup of my favourite programs I’d waste my time searching the icons or menus.

As Xfce doesn’t provide me with the powerful one, I use: Synapse.

File manager

Well, there are times when I play with files in the command line, but sometimes it just fits better to use some UI for the purpose.

My choice is Krusader. Well, the most stuff I need is available, namely:

  • file/folder manipulation,
  • file folder comparison and
  • file/folder contents search.

The only thing that bothers me is the fact that development is rather stalled in the project.

Still, viable alternative might be: Double commander, which seem to be even cross-platform and can use the total commander plugins (which used to be my choice on windows for the purpose).

Editor

I tried to live with Gedit and for simple note-taking it might be a good choice, however as I like to play with ruby these days, I tried to find something that would help more in the area.

(G)Vim

After searching I came to conclusion, that (G)Vim is quite popular in the Linux world.

Well, it’s worth to mention vim joke here, which expresses the feelings of many on this editor: Infinite Vim monkey theorem (see Infinite monkey theorem for explanation).

Still, I’ve seen:

  • many people favouring it,
  • it’s the first editor I remember to have available once logging in to my school linux account (OK no real argument, just sentiment),
  • it seemed to provide countless plugins for all sorts of stuff and
  • it’s incredibly powerful, as far as I’ve seen and learning it might pay back.

So I decided cca 2 years back to uninstall the editors I’ve been used to and force myself to use (G)Vim.

Great source of information was for me: vimcasts, which helped me a lot in the area.

Well, I plan to document my .vimrc setup in a separate post, as it could make this one way too long.

Files sync tool

Unison is my choice her. For more info on my use case, see this blog post.

Conclusion

Hope you find some inspiration here. And would be glad to hear from you guys about any I missed, but you could not live without.

Still, I can’t believe anyone read this far. As I guess I would not force myself to :).

Friday, July 11, 2014

Kindle for offline html docs reading

Summer is here, while laying on the beach, you can use the time to learn new programming language or get some overview on the framework of your interest. Possibly the things you had no time left for over the year.

I bought myself Kindle, my beach holiday is just a day away, so it’s time to fill the reader with content.

Goal

Get an online html docs to Kindle for offline reading.

Required software

  • wget - download program
  • calibre - ebook reader/converter/…

Process

  1. find the docs you’re interested in (I’ll refer to <http://your-manual.com> as the url of the docs of my interest)
  2. download it (I used wget to do so, inspired by stackoverflow answer, please note this is expecting all docs no one domain only):

    wget --recursive --timestamping -l inf --no-remove-listing --page-requisites <http://your-manual.com>
  3. start calibre
  4. drag&drop the downloaded index (main) page to calibre
  5. convert to mobi format (via right click on the book and choosing Convert Books -> Convert individually afterwards editing metadata and choosing mobi as the Output format)
  6. connect your kindle (via usb)
  7. send ebook to device - via right clicking the ebook and choosing: Send to device -> Send to main memory
  8. done!

The valuable part

Well, the challenging part is for me always to actually digest the docs. So go ahead and read it (if it’s worth it). Self-discipline, I need to work on that, I guess.

Wednesday, June 25, 2014

Migrating from bash to zsh

These are my steps in migration from bash to zsh. I’m documenting these to keep the future reference for me (and possibly other readers as well).

Motivation?

Check out the Slideshare presentation: Why Zsh is Cooler than Your Shell

Current shell

to find out the current shell:

ps -p $$

it’s useful during migration to make sure the new shell is really in use.

Migration

Migration itself included (on my Fedora) reconfiguring:

oh-my-zsh

There exists users’ maintained collection of zsh customizations, that I found useful. Called: oh-my-zsh. (Please note the amazing Github’s - Star count. Once seeing that I didn’t doubt I’ll give it a try.)

For installation I just went for:

curl -L http://install.ohmyz.sh | sh

oh-my-zsh with powerline

I’m used to powerline everywhere ((g)vim/konsole/tmux/bash) already, so let’s keep it working in zsh as well.

I created file: ~/.oh-my-zsh/custom/powerline.zsh with contents:

. ~/.local/lib/python2.7/site-packages/powerline/bindings/zsh/powerline.zsh

Please note: You need to go for the actual location of powerline installation on your system.

and in the file: ~/.zshrc I had to comment out line:

# ZSH_THEME="robbyrussell"

otherwise oh-my-zsh theme seems to conflict with powerline one.

oh-my-zsh plugins

The real benefit of oh-my-zsh commes with plugins, so I went for those I find useful. Via changing the file: ~/.zshrc having:

plugins=(git mvn glassfish yum colored-man vagrant z common-aliases gradle)

oh-my-zsh custom stuff

As my ~/.bashrc held already quite some customizations, I migrated those to:

~/.oh-my-zsh/custom/*.zsh

Even this part was nice, as I could tidy up things a bit and omit unused stuff. Moreover I decided to create multiple *.zsh files (as these are auto-loaded) to achieve modularity.

Done

Honestly in the beginning, I had no clue how much I would enjoy my new shell.

Friday, June 13, 2014

Xfce: How to disable toolbar text in KDE apps

Recently I’ve experienced quite a funny situation. I use Krusader for file browsing that calls Kompare for file comparisons. However I was not able to copy changes from left to right (or vice versa) in the compared files, as I could not see all the toolbar buttons (these were too big). I didn’t even notice for quite some time, that there are others, not displayed on my screen.

But who needs text description in toolbars? Well, I don’t!

I remembered global setting from the Gnome world that it should be possible, so searched from the KDE equivalent.

Found it (in my Fedora + Xfce) in:

Xfce menu button -> Options (hope this one is right, as I have it as Einstellungen) -> KDE SystemOptions (for me KDE SystemEinstellungen) -> Style (Stil) -> Fine Tuning tab -> Toolbars section
where I just chose No Text in the both: Main toolbar text as well as in the Secondary toolbar text.

That did the job for me.

Apache Ant tasks for JMX access

I wanted to invoke JMX operations from the Ant tasks. However finding usable ant tasks library as well as the usage was rather tricky. So let me share my experience to make things easier for others.

Ant tasks for JMX operations

I decided to follow Tomcat documentation and used ant tasks distributed with tomcat.

Just for the record the usage is not restricted to Tomcat deployed JMX mBeans. For me it worked for java process accessible via JConsole via Remote connection.

Retrieving the library

  1. As I wanted to get the latest version I used maven central repository “search by classname” feature and searched for: org.apache.catalina.ant.jmx.JMXAccessorTask (see the query)
  2. afterwards I went for the tomcat 8 jar file (called tomcat-catalina-ant-8.0.8.jar)
  3. and just copyied the latest available to my $ANT_HOME/lib dir.

Usage

I didn’t have a chance (or motivation?) to check all the tasks available, the full list of tasks available can be seen in the zipped file: org/apache/catalina/ant/jmx/antlib.xml, following were present for me:

<typedef
    name="open"
    classname="org.apache.catalina.ant.jmx.JMXAccessorTask" />
<typedef
    name="set"
    classname="org.apache.catalina.ant.jmx.JMXAccessorSetTask" />
<typedef
    name="get"
    classname="org.apache.catalina.ant.jmx.JMXAccessorGetTask" />
<typedef
    name="invoke"
    classname="org.apache.catalina.ant.jmx.JMXAccessorInvokeTask" />
<typedef
    name="query"
    classname="org.apache.catalina.ant.jmx.JMXAccessorQueryTask" />
<typedef
    name="create"
    classname="org.apache.catalina.ant.jmx.JMXAccessorCreateTask" />
<typedef
    name="unregister"
    classname="org.apache.catalina.ant.jmx.JMXAccessorUnregisterTask" />
<typedef
    name="equals"
    classname="org.apache.catalina.ant.jmx.JMXAccessorEqualsCondition" />
<typedef
    name="condition"
    classname="org.apache.catalina.ant.jmx.JMXAccessorCondition" />

out of these, I gave following a try:

org.apache.catalina.ant.jmx.JMXAccessorTask
org.apache.catalina.ant.jmx.JMXAccessorInvokeTask
org.apache.catalina.ant.jmx.JMXAccessorQueryTask

For the demonstration purposes I’m using Glassfish 4.0.

Example: Listing JMX MBeans

Let’s assume we want to retrieve the MBean by name (namely: java.lang:type=Memory). Please note username and password were not required for access (otherwise they should be specified via respective properties).

Noteworthy here is the resultproperty, which could hold array from which we could get a name. So having in ant build script:

<typedef 
    name="jmxQuery"
    classname="org.apache.catalina.ant.jmx.JMXAccessorQueryTask" />

<jmxQuery
    host="localhost"
    port="8686"
    echo="true"
    name="java.lang:type=Memory"
    resultproperty="memory" />

<echo>Retrieved MBeans count: ${memory.Length}</echo>
<echo>The 1.st one has name: ${memory.0.Name}</echo>

results for me in following output:

 [jmxQuery] memory.Length=1
 [jmxQuery] memory.0.Name=java.lang:type=Memory
     [echo] Retrieved MBeans count: 1
     [echo] The 1.st one has name: java.lang:type=Memory

Example: Invoking operation via JMX

Here is a 2 step approach required:

  1. connect to remote server via JMX and afterwards
  2. invoke the operation on the particular MBean.

For demonstration purposes, let’s assume we want to call garbage collection (via invoking operation: gc() on MBean named: java.lang:type=Memory)

Sample ant build file chunk does the job (please note ref property value that has to be the same across these 2 tasks):

<typedef 
    name="jmxOpen"
    classname="org.apache.catalina.ant.jmx.JMXAccessorTask" />

<typedef 
    name="jmxInvoke"
    classname="org.apache.catalina.ant.jmx.JMXAccessorInvokeTask" />

<jmxOpen
    host="localhost"
    port="8686"
    ref="glassfish"
    failOnError="true" />

<jmxInvoke
    name="java.lang:type=Memory"
    operation="gc"
    echo="true"
    delimiter=" "
    failOnError="true"
    ref="glassfish" />

Further information

For more details, I recommend reading the official Tomcat documentation as well as Javadocs.

Thursday, June 12, 2014

Using Genius TVGo DVB-T03 on Fedora

My other laptop was not available and the start of Football WM 2014 was just less than 30 minutes away. Let’s see if I can watch it on my laptop with Fedora (16) with my Genius TVGo DVB-T03 device.

The most of the steps were inspired by fedora wiki: https://fedoraproject.org/wiki/DavidTimms/DVB

DVB-T required package

I installed dvb-apps package via:

sudo yum install dvb-apps

Afterwards checked the file list in package via:

rpm -q --fileprovide dvb-apps

which contained also:

/usr/bin/scandvb

(please note we’ll need this one later).

Firmware

I plugged in my dvb-t usb and seen after running dmesg:

[11140.318728] usb 2-1.2: new high-speed USB device number 6 using ehci-pci
[11140.397936] usb 2-1.2: New USB device found, idVendor=0458, idProduct=4012
[11140.397946] usb 2-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[11140.397952] usb 2-1.2: Product: DVB-T 2
[11140.397956] usb 2-1.2: Manufacturer: Afatech
[11140.557325] usbcore: registered new interface driver dvb_usb_af9015
[11140.557753] usb 2-1.2: dvb_usb_v2: found a 'Genius TVGo DVB-T03' in cold state
[11140.557900] usb 2-1.2: dvb_usb_v2: Did not find the firmware file 'dvb-usb-af9015.fw'. Please see linux/Documentation/dvb/ for more details on firmware-problems. Status -2
[11140.557905] usb 2-1.2: dvb_usb_v2: 'Genius TVGo DVB-T03' error while loading driver (-2)

so searched a bit and found + downloaded rpm from: http://pkgs.org/fedora-centos-rhel-opensuse-mandriva/olea/dvb-usb-af9015-fw-4.95.0-1.noarch.rpm.html

Afterwards installed the downloaded firmware via:

sudo rpm -ivh dvb-usb-af9015-fw-4.95.0-1.noarch.rpm

And finally unplugged and plugged to usb again to see:

[11796.948500] usb 2-1.1: new high-speed USB device number 7 using ehci-pci
[11797.038703] usb 2-1.1: New USB device found, idVendor=0458, idProduct=4012
[11797.038714] usb 2-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[11797.038720] usb 2-1.1: Product: DVB-T 2
[11797.038725] usb 2-1.1: Manufacturer: Afatech
[11797.041521] usb 2-1.1: dvb_usb_v2: found a 'Genius TVGo DVB-T03' in cold state
[11797.042331] usb 2-1.1: dvb_usb_v2: downloading firmware from file 'dvb-usb-af9015.fw'
[11797.119176] usb 2-1.1: dvb_usb_v2: found a 'Genius TVGo DVB-T03' in warm state
[11797.542424] usb 2-1.1: dvb_usb_v2: will pass the complete MPEG2 transport stream to the software demuxer
[11797.542542] DVB: registering new adapter (Genius TVGo DVB-T03)
[11797.575204] i2c i2c-7: af9013: firmware version 4.95.0.0
[11797.578207] usb 2-1.1: DVB: registering adapter 0 frontend 0 (Afatech AF9013)...
[11797.589973] tda18271 7-00c0: creating new instance
[11797.597341] TDA18271HD/C1 detected @ 7-00c0
[11797.802656] Registered IR keymap rc-empty
[11797.802848] input: Genius TVGo DVB-T03 as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/rc/rc0/input12
[11797.803814] rc0: Genius TVGo DVB-T03 as /devices/pci0000:00/0000:00:1d.0/usb2/2-1/2-1.1/rc/rc0
[11797.803825] usb 2-1.1: dvb_usb_v2: schedule remote query interval to 500 msecs
[11797.803832] usb 2-1.1: dvb_usb_v2: 'Genius TVGo DVB-T03' successfully initialized and connected

so firmware seems to work now.

Generating channels.conf

Following command did the job for me (as I’m located in Bayern):

scandvb /usr/share/dvb/dvb-t/de-Bayern  > channels.conf

Enjoying the broadcasts

There is no me-tv (that I’m used to it from Ubuntu world), still, there is a vlc, that does the job for me quite well. Just running:

vlc channels.conf

made it and I could watch the game!

Friday, June 6, 2014

Eclipse plugin for svn integration

I’ve found it tricky to install Eclipse plugin providing svn integration.

The problem is:

  • which plugin to choose,
  • which version of the plugin to choose as well as
  • which packages to install.

I’m running latest Eclipse (Kepler SR2) and plugin of my choice is subclipse.

The Eclipse plugin update site has to be chosen based on the svn version installed on the machine (check it running svn --version):

Following packages were required for installation (for me):

  • Subclipse (Required)
  • SVNKit Client Adapter (Not Required)
  • SVNKit Library

That’s it! As it covers my Eclipse/svn day-to-day workflow.

Tuesday, March 4, 2014

Markdown editor for blogger

Markdown is quite popular these days and I like it for it’s simplicity. So I’ve been searching for a way to integrate it in my blog writing workflow. In Blogger, it’s unfortunately not an option.

There I could go:

  • either for WYSIWYG editor or
  • manual HTML writing.

However WYSIWYG generates too much HTML I don’t need. Moreover some pre-publish manual HTML editing is a pain then. And the other option I consider it rather slow/unproductive.

So, I looked for other options, and found stackedit.io. Even this post is (and couple others as well) written in it.

stackedit

I consider Stackedit great as it:

  • is Markdown editor,
  • is open source (see github repo),
  • has awesome UI (including live preview, key bindings, …),
  • has integration with other popular services that I use anyway:
  • has active community (just check the stars count and commit activity on the github repo) and
  • has impressive feedback time (resolution on my issues/questions came in couple hours)

So my post writing/editing workflow goes like this:

  1. write/edit post on stackedit.io,
  2. sync it to google drive and
  3. publish/republish it to blogger.

That’s it! No further in-blogger updates required!

Blogger specifics

Still there are some specifics in my workflow (to provide smooth blogger integration).

  • using the Interpreted variables: title and tags via:
---
title: Markdown editor for blogger
tags: blogger markdown stackedit.io
---

p6spy 2.0.0 is out!

After 8 years P6spy came to it’s next stable release!
You can get it here.

The last stable release happened (based on maven central repo) on 27-Dec-2005 (1.3 version). That is quite some time, so one would expect many things to happen in the meantime. Well the truth is that project was (half) dead for quite some time.

Changes

I can’t comment on full history since 1.3 release (since my interest in project started last summer), still I’ve noticed following:

  • project hosting was moved from sourceforge to github,
  • major part of the legacy code was refactored,
  • Java 6/7 JDBC API support introduced,
  • proxying via modified JDBC URLs only was implemented,
    So for for MySQL original url would be:

    jdbc:mysql://<hostname>:<port>/<database>

    the one proxied via p6spy would one:

    jdbc:p6spy:mysql://<hostname>:<port>/<database>

    without a need for any further configuration,

  • XA Datasource support has been introduced,
  • configuration via:
    • system/environment properties and
    • JMX properties
    • as an alternative to file configuration only
    • or even zero config use case supported,
  • slf4j support (more flexible as previously used log4j),
  • junit tests were migrated to junit 4 (well most of the old ones were failing anyway),
  • Continuous integration using Travis was setup providing testing on popular:
    • DB systems (namely: Oracle, DB2, PostgreSQL, MySQL, H2, HSQLDB, SQLite, Firebird, and Derby), see build status on: travis-ci as well as
    • application servers (namely: Wildfly 8, JBoss 4.2, 5.1, 6.1, 7.1, Glassfish 3.1, 4.0, Jetty 7.6, 8.1, 9.1, Tomcat 6, 7, 8, Resin 4, Jonas 5.3 and Geronimo 2.1, 2.2), see build status on: travis-ci.

Full changelog

For the full changelog, see issues fixed in: 2.0.0-alpha1 as well as 2.0.0.

Postgres and Oracle compatibility with Hibernate

There are situations your JEE application needs to support Postgres and Oracle as a Database.
Hibernate should do the job here, however there are some specifics worth mentioning.
While enabling Postgres for application already running Oracle I came across following tricky parts:

  • BLOBs support,
  • CLOBs support,
  • Oracle not knowing Boolean type (using Integer) instead and
  • DUAL table.

These were the tricks I had to apply to make the @Entity classes running on both of these.

Please note I’ve used Postgres 9.3 with Hibernate 4.2.1.SP1.

BLOBs support

The problem with Postgres is that it offers 2 types of BLOB storage:

  • bytea - data stored in table
  • oid - table holds just identifier to data stored elsewhere

I guess in the most of the situations you can live with the bytea as well as I did. The other one as far as I’ve read is to be used for some huge data (in gigabytes) as it supports streams for IO operations.

Well, it sounds nice there is such a support, however using Hibernate in this case can make things quite problematic (due to need to use the specific annotations), especially if you try to achieve compatibility with Oracle.

To see the trouble here, see StackOverflow: proper hibernate annotation for byte[]

All- the combinations are described there:

annotation                   postgres     oracle      works on
-------------------------------------------------------------
byte[] + @Lob                oid          blob        oracle
byte[]                       bytea        raw(255)    postgresql
byte[] + @Type(PBA)          oid          blob        oracle
byte[] + @Type(BT)           bytea        blob        postgresql

where @Type(PBA) stands for: @Type(type="org.hibernate.type.PrimitiveByteArrayBlobType") and @Type(BT) stands for: @Type(type="org.hibernate.type.BinaryType").

These result in all sorts of Postgres errors, like:

ERROR: column “foo” is of type oid but expression is of type bytea

or

ERROR: column “foo” is of type bytea but expression is of type oid

Well, there seems to be a solution, still it includes patching of Hibernate library (something I see as the last option when playing with 3.rd party library).

There is also a reference to official blog post from the Hibernate guys on the topic: PostgreSQL and BLOBs. Still solution described in blog post seems not working for me and based on the comments, seems to be invalid for more people.

BLOBs solved

OK, so now the optimistic part.

After quite some debugging I ended up with the Entity definition like this :

@Lob
private byte[] foo;

Oracle has no trouble with that, moreover I had to customize the Postgres dialect in a way:

public class PostgreSQLDialectCustom extends PostgreSQL82Dialect {

    @Override
    public SqlTypeDescriptor remapSqlTypeDescriptor(SqlTypeDescriptor sqlTypeDescriptor) {
    if (sqlTypeDescriptor.getSqlType() == java.sql.Types.BLOB) {
      return BinaryTypeDescriptor.INSTANCE;
    }
    return super.remapSqlTypeDescriptor(sqlTypeDescriptor);
  }
}

That’s it! Quite simple right? That works for persisting to bytea typed columns in Postgres (as that fits my usecase).

CLOBs support

The errors in misconfiguration looked something like this:

org.postgresql.util.PSQLException: Bad value for type long : ...

So first I’ve found (on String LOBs on PostgreSQL with Hibernate 3.6) following solution:

@Lob
@Type(type = "org.hibernate.type.TextType")
private String foo;

Well, that works, but for Postgres only.

Then there was a suggestion (on StackOverflow: Postgres UTF-8 clobs with JDBC) from to go for:

@Lob
@Type(type="org.hibernate.type.StringClobType")
private String foo;

That pointed me the right direction (the funny part was that it was just a comment to some answers). It was quite close, but didn’t work for me in all cases, still resulted in errors in my tests.

CLOBs solved

The important was @deprecation javadocs in the org.hibernate.type.StringClobType that brought me to working one:

@Lob
@Type(type="org.hibernate.type.MaterializedClobType")
private String foo;

That works for both Postgres and Oracle, without any further hacking (on Hibernate side) needed.

Boolean type

Oracle knows no Boolean type and the trouble is that Postgres does. As there was also some plain SQL present, I ended up In Postgres with error:

ERROR: column “foo” is of type boolean but expression is of type integer

I decided to enable cast from Integer to Boolean in Postgres rather than fixing all the plain SQL places (in a way found in Forum: Automatically Casting From Integer to Boolean):

update pg_cast set castcontext = 'i' where oid in ( select c.oid from pg_cast c inner join pg_type src on src.oid = c.castsource inner join pg_type tgt on tgt.oid = c.casttarget where src.typname like 'int%' and tgt.typname like 'bool%');

Please note you should run the SQL update by user with provileges to update catalogs (probably not your postgres user used for DB connection from your application), as I’ve learned on Stackoverflow: Postgres - permission denied on updating pg_catalog.pg_cast.

DUAL table

There is one more specific in the Oracle I came across. If you have plain SQL, in Oracle there is DUAL table provied (see more info on Wikipedia on that) that might harm you in Postgres.

Still the solution is simple. In Postgres create a view that would fill the similar purpose. It can be created like this:

create or replace view dual as select 1;

Conclusion

Well that should be it. Enjoy your cross DB compatible JEE apps.

Tuesday, January 21, 2014

yeoman - testing with mocha and chai instead of jasmine

I've been playing around with yeoman these days. It seems like project worth a try for my free time single page application.

After the inital setup (using AngularJS generator) I decided to let my tests run with mocha + chai (with expect asertions) rather than jasmine (generated by default).

In fact it was super-simple. The diffs that matter:
diff --git a/yo/karma.conf.js b/yo/karma.conf.js
index bc0e168..385045f 100644
--- a/yo/karma.conf.js
+++ b/yo/karma.conf.js
@@ -7,7 +7,7 @@ module.exports = function(config) {
     basePath: '',
 
     // testing framework to use (jasmine/mocha/qunit/...)
-    frameworks: ['jasmine'],
+    frameworks: ['mocha', 'chai'],
 
     // list of files / patterns to load in the browser
     files: [
diff --git a/yo/package.json b/yo/package.json
index 3e3db14..9649a55 100644
--- a/yo/package.json
+++ b/yo/package.json
@@ -34,7 +34,8 @@
     "karma-html2js-preprocessor": "~0.1.0",
     "karma-firefox-launcher": "~0.1.3",
     "karma-script-launcher": "~0.1.0",
-    "karma-jasmine": "~0.1.5",
+    "karma-mocha": "~0.1.1",
+    "karma-chai": "~0.0.2",
     "karma-coffee-preprocessor": "~0.1.2",
     "requirejs": "~2.1.10",
     "karma-requirejs": "~0.2.1",
diff --git a/yo/test/spec/controllers/main.js b/yo/test/spec/controllers/main.js
index eb7cda2..f9becde 100644
--- a/yo/test/spec/controllers/main.js
+++ b/yo/test/spec/controllers/main.js
@@ -17,6 +17,6 @@ describe('Controller: MainCtrl', function () {
   }));
 
   it('should attach a list of awesomeThings to the scope', function () {
-    expect(scope.awesomeThings.length).toBe(3);
+    expect(scope.awesomeThings.length).to.equal(3);
   });
 });
Afterwards just running (to fetch missing packages):
npm install
and tests succeed:
grunt test

Tuesday, December 17, 2013

Liquibase - implementing custom type.

There are some things to keep in mind while implementing custom type in liquibase. Let's see how to create one.

Existing database types

Well, this might be the biggest advantage of using/hacking the open source projects. You have a chance to see how are things done. This is useful to find your inspiration, the places that require modifications and the hooks to existing system.

Classes worth checking for our purposes are in the package:
liquibase.datatype.core
You can browse them directly on github.

Sample custom BlobType implementation

Let's assume we're about to modify BlobType. Our implementation could look like this:
package liquibase.datatype.core;

import liquibase.database.Database;
import liquibase.database.core.PostgresDatabase;
import liquibase.datatype.DataTypeInfo;
import liquibase.datatype.DatabaseDataType;
import liquibase.datatype.LiquibaseDataType;

@DataTypeInfo(name = "blob", aliases = { "longblob", "longvarbinary", "java.sql.Types.BLOB",
    "java.sql.Types.LONGBLOB", "java.sql.Types.LONGVARBINARY", "java.sql.Types.VARBINARY",
    "varbinary" }, minParameters = 0, maxParameters = 0, priority = LiquibaseDataType.PRIORITY_DATABASE)
public class BlobTypeTest extends BlobType {
  
  public DatabaseDataType toDatabaseDataType(Database database) {
    // handle the specifics here, you can go for the per DB specifics, let's assume Postgres
    if (database instanceof PostgresDatabase) {
       // your custom type here
    }
    // use defaults for all the others
    return super.toDatabaseDataType(database);
  }

}

Specifics to keep in mind

There are some specifics that should be considered:
  • DataType priority is important

    To make sure our type will be considered in favor to default implementation. We're going for:
    priority = LiquibaseDataType.PRIORITY_DATABASE
    
    where the default one (in the supertype) is:
    priority = LiquibaseDataType.PRIORITY_DEFAULT
    
    This just means that our implementation should be considered in favor to default one.

    See method:
    liquibase.datatype.DataTypeFactory.register()
    
    implementation for details.
  • DataType registration considers specific packages to be scanned only

    We have more options here, but our stuff should go to any of these:
    • any of those listed in jar/MANIFEST.MF property:
      Liquibase-Package
      
      Where the default set in the liquibase-core-3.0.8.jar is:
      Liquibase-Package: liquibase.change,liquibase.database,liquibase.parse
       r,liquibase.precondition,liquibase.datatype,liquibase.serializer,liqu
       ibase.sqlgenerator,liquibase.executor,liquibase.snapshot,liquibase.lo
       gging,liquibase.diff,liquibase.structure,liquibase.structurecompare,l
       iquibase.lockservice,liquibase.ext
      
    • comma separated custom package list provided via system property called:
      liquibase.scan.packages
      
    • if all of the above are empty, note the fallback package list is used. As implementation says:
      if (packagesToScan.size() == 0) {
      	addPackageToScan("liquibase.change");
      	addPackageToScan("liquibase.database");
      	addPackageToScan("liquibase.parser");
      	addPackageToScan("liquibase.precondition");
      	addPackageToScan("liquibase.datatype");
      	addPackageToScan("liquibase.serializer");
      	addPackageToScan("liquibase.sqlgenerator");
      	addPackageToScan("liquibase.executor");
      	addPackageToScan("liquibase.snapshot");
      	addPackageToScan("liquibase.logging");
      	addPackageToScan("liquibase.diff");
      	addPackageToScan("liquibase.structure");
      	addPackageToScan("liquibase.structurecompare");
      	addPackageToScan("liquibase.lockservice");
      	addPackageToScan("liquibase.ext");
      }
      

    See method:
    ServiceLocator.setResourceAccessor()
    
    implementation for details.

    Well, as you might have noticed, I'm lazy enough as I went for the already registered package:
    liquibase.datatype.core
    
    so it worked once my implementation is on the classpath.
That should be it.

Debugging ant task

It's allways good idea to debug once playing around with the custom changes in the 3.rd party code.
As I went for ant task, just addopted ANT_OPTS variable. In my case (as I'm on linux) following worked:
export ANT_OPTS="-Xdebug -agentlib:jdwp=transport=dt_socket,server=y,address=8000"
remote debugging was possible afterwards.

To check if custom type is registered, check in the debug session the constructor:
DataTypeFactory.DataTypeFactory()
local variable "classes" contents after line:
classes = ServiceLocator.getInstance().findClasses(LiquibaseDataType.class);
to see all the types found.

That should provide you the basics on liquibase types hacking.

Monday, December 16, 2013

Oracle JDK 1.7.0_u45 incompatibility with Glassfish 3.x (Corba)

Q: Did you give a try to Glassfish 3.1.x with latest stable JDK (1.7.0_u45)?
A: Well, you might be interested in possible trouble there.

After quite some debugging of weird error happening, namely:
Caused by: java.lang.ArrayIndexOutOfBoundsException: -1636631191
at java.util.HashMap.put(HashMap.java:498)
at java.util.HashSet.add(HashSet.java:217)
...
I ended up creating: https://java.net/jira/browse/GLASSFISH-20927 (Btw, it was quite weird, but I'm not allowed to add attachments in their jira :)
The problem seems to be in internals of the HashSet -> HashMap. Where Corba doesn't seem to set the expected reference to EMPTY_TABLE in case of empty Set. Let's see how they proceed with the analysis.

Tuesday, October 22, 2013

My Firefox browsing privacy settings

I decided to go for a bit more privacy for daily browsing. I don't think I need full security (I'm active in social networks anyway, so that would be the 1.st thing to cut), but let's say at least some. I've been influenced of the duckduckgo's donttrack.us and fixtracking.com pages.

I'm running Linux, so for the browser I have 2 mainstream options:
  • Chrome/Chromium
  • Firefox
The Chrome throws
Segmentation fault (core dumped)
at me (on both of my linux machines - Xubuntu/Fedora). Due to lack of motivation I didn't investigate any further (possibly some misbehaving extension). So my options narrow to Firefox only. But that is OK with me. As I'm used to it after some years of usage.

Firefox setup

I'm using duckduckgo.com as my default search engine in both
  • Location bar:
    • To set it, go to page:
      about:config 
      
      and set property:
      keyword.URL 
      
      to:
      https://duckduckgo.com/?q=
      
    and
  • Search bar:
    • To set it click down arrow in the Search bar and choose "Manage Search Engines ..." where move DuckDuckGo to the very top
So I can escape from search + profile association promoting me next day all the stuff I searched for a couple days ago.

Add-ons

Moreover I go for the following add-ons:

Unsecure part

Well there are some specifics in my setup. As mentioned earlier I'm using social networks for posting, and as I'm lazy enough and like posting with minimal effort, I use also ShareThis. However since installing all the previously mentioned, ShareThis stopped working. Following are the tweaks I had to do to enable it again:
  • DoNoTrackMe - I clicked on toolbar icon and then options wheel in the corner, where I disabled "ShareThis" blocking,
  • moreover as Sharethis page rendered on sharing has problems with HTTPs certificate, I had to deactivate it in the - HTTPS Everywhere - as well.
  • Once I'm ready to share site and click Sharethis button in location bar (nothing happens so) I have to go for "Disconnect me" toolbar button and choose "Advertising" category and uncheck "Sharethis". I need to click Sharethis button once again then.
That's it for my setup. I don't think I improved my privacy in a radical way, but let's say I went for options that don't hurt my common browsing experience that much. Some (like ads blocking) make it even more pleasant.

Eclipse toolbar housekeeping

It took me quite some time, since I really started to care about all the wasted space in my IDE of choice - Eclipse IDE. The problem is that toolbar contains elements that I never use. Reasons might differ. Some I don't need, for some I use keyboard shortcuts (that I consider faster to use) for the rest I don't have a clue what they're intended for.

Once I checked the options, motivated by the removal of "Quick access" element eating my toolbar space (since Eclipse Juno version) I've found the stackoverflow post providing even more help than I originally expected.

It correctly pointed me to yet unresolved Eclipse bug, however that was not what I was looking for. Still there were other answers giving me what I came for.

Removing "Quick access" element

As this answer pointed, adding:
#SearchField {
   visibility:hidden;
}
to the:
<ECLIPSE_HOME>/plugins/org.eclipse.platform_<VERSION>/css/e4_basestyle.css
and restarting Eclipse does the job.

Removing toolbar buttons

As the other answer pointed, going for: Window -> Customize Prespective ... I was able to get rid of the buttons (I don't need) in my toolbar.
Please note: this should be done on per perspective basis.

Conclusion

Since applying the mentioned, my toolbar behaves and takes one row of my vertical space only. In fact, it's like half empty :)

Sunday, October 20, 2013

How I left the output redirection to file.

If you feel with linux command line at home, there is often a situation you pipe command outputs. These enable us doing amazing things.

However there are times, when console is not the right place to examine the final output and your favourite editor could do much better job.
In the past I used to output to file and open afterwards, until,... I've noticed it's possible to pipe directly to editor.

Let's assume we're interested in reading tail output in our editor of choice.

Gedit

As my default editor used to be Gedit for quite some time, let's see it in action:
tail -f some.log | gedit
No surprise :), I guess. No extra arguments. Nice!

Morever gedit even provides me with the indicatior (displayed on the tab), that loading is in progress and automatically refreshes contents.

Gvim

I'm in process of transition to gvim usage. I've heard about it's power, never really had a chance to dive deep there. However after watching some vimcasts and reading couple reviews, I'm quite amazed. I'm still trying to memorize the keys for specific tasks (that should be just a question of time and frequency of usage of particular ones).

I made it already my primary editor (wherever I used to use Gedit before).

OK, let's see it in action:
tail -f some.log | gvim -R -
please note -R is not mandatory, but useful, as it indicates file beeing read only.

Kate

I'm not really a KDE guy (these days/years :), but this might be be still be useful for those using kate editor:
tail -f some.log | kate -i 

Bash integration

In the comments of the sites (commandlinefu), where I've found the solution for gvim was one more usecase that caught my attention. Having in the .bashrc following:
function gv() {
  $@ | gvim -R -
}
Enables me doing:
gv tail -f some.log
what makes things even more comfortable. Sure you could replace the function name, as well as the editor with the ones you prefer.

Well, that's it, enjoy (or forget it if you find it useless :) or if interested, share how you achieve it with your editor of choice.

XML processing in shell.

Q: Have you ever been struggling with xml processing in shell?
A: There's an elegant tool: Xmlstarlet (http://xmlstar.sourceforge.net/).

Why would you care?

Well, if you:
  • use the shell environment (bash in my case),
  • have a need for XML data extraction/transformation and
  • you know/are willing to learn XPath/Xslt
then you should. In fact it might be the perfect match.

Download/Installation

Follow the official docs. According to the link, if you're Linux (it's easy as things should be, I guess): "Bundled with your nearest Linux distribution"

Usage

Well, I'd recommend to check the:
man xmlstarlet
and
xmlstarlet --help
I'm going to focus on the one part of the functionality only, namely: data extraction, to see the official help, go for:
xmlstarlet sel
Output is quite impressive, giving you even some examples, for those lazy (like me), I copy/paste here:
XMLStarlet Toolkit: Select from XML document(s)
Usage: xmlstarlet sel <global-options> {<template>} [ <xml-file> ... ]
where
  <global-options> - global options for selecting
  <xml-file> - input XML document file name/uri (stdin is used if missing)
  <template> - template for querying XML document with following syntax:
 
<global-options> are:
  -Q or --quiet             - do not write anything to standard output.
  -C or --comp              - display generated XSLT
  -R or --root              - print root element <xsl-select>
  -T or --text              - output is text (default is XML)
  -I or --indent            - indent output
  -D or --xml-decl          - do not omit xml declaration line
  -B or --noblanks          - remove insignificant spaces from XML tree
  -E or --encode <encoding> - output in the given encoding (utf-8, unicode...)
  -N <name>=<value>         - predefine namespaces (name without 'xmlns:')
                              ex: xsql=urn:oracle-xsql
                              Multiple -N options are allowed.
  --net                     - allow fetch DTDs or entities over network
  --help                    - display help
 
Syntax for templates: -t|--template <options>
where <options>
  -c or --copy-of <xpath>   - print copy of XPATH expression
  -v or --value-of <xpath>  - print value of XPATH expression
  -o or --output <string>   - output string literal
  -n or --nl                - print new line
  -f or --inp-name          - print input file name (or URL)
  -m or --match <xpath>     - match XPATH expression
  --var <name> <value> --break or
  --var <name>=<value>      - declare a variable (referenced by $name)
  -i or --if <test-xpath>   - check condition <xsl:if test="test-xpath">
  --elif <test-xpath>       - check condition if previous conditions failed
  --else                    - check if previous conditions failed
  -e or --elem <name>       - print out element <xsl:element name="name">
  -a or --attr <name>       - add attribute <xsl:attribute name="name">
  -b or --break             - break nesting
  -s or --sort op xpath     - sort in order (used after -m) where
  op is X:Y:Z,
      X is A - for order="ascending"
      X is D - for order="descending"
      Y is N - for data-type="numeric"
      Y is T - for data-type="text"
      Z is U - for case-order="upper-first"
      Z is L - for case-order="lower-first"
 
There can be multiple --match, --copy-of, --value-of, etc options
in a single template. The effect of applying command line templates
can be illustrated with the following XSLT analogue
 
xml sel -t -c "xpath0" -m "xpath1" -m "xpath2" -v "xpath3" 
        -t -m "xpath4" -c "xpath5"
 
is equivalent to applying the following XSLT
 
<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:template match="/">
  <xsl:call-template name="t1"/>
  <xsl:call-template name="t2"/>
</xsl:template>
<xsl:template name="t1">
  <xsl:copy-of select="xpath0"/>
  <xsl:for-each select="xpath1">
    <xsl:for-each select="xpath2">
      <xsl:value-of select="xpath3"/>
    </xsl:for-each>
  </xsl:for-each>
</xsl:template>
<xsl:template name="t2">
  <xsl:for-each select="xpath4">
    <xsl:copy-of select="xpath5"/>
  </xsl:for-each>
</xsl:template>
</xsl:stylesheet>
 
XMLStarlet is a command line toolkit to query/edit/check/transform
XML documents (for more information see http://xmlstar.sourceforge.net/)
 
Current implementation uses libxslt from GNOME codebase as XSLT processor
(see http://xmlsoft.org/ for more details)
Please note the switch:
-C or --comp              - display generated XSLT
This can really help if you're struggling on particular problem, to see what's really done in the background.

Let's jump directly to examples.

XmlStarlet in action

Imagine sample xml (copied from http://www.w3schools.com/xml/xml_attributes.asp) called sample.xml:
<messages>
  <note id="501">
    <to>Tove</to>
    <from>Jani</from>
    <heading>Reminder</heading>
    <body>Don't forget me this weekend!</body>
  </note>
  <note id="502">
    <to>Jani</to>
    <from>Tove</from>
    <heading>Re: Reminder</heading>
    <body>I will not</body>
  </note>
</messages>


Element value selection

Let's say, we want to extract all 'from' element values. Using:
xmlstarlet sel -t -m "messages" -m "note" -v "from" < sample.xml
we get:
JaniTove
So what have we done?
  • sel - select data or query XML document
  • -t - template definition
  • -m "messages" - match "messages" XPath expression
  • -m "note" - match "note" XPath expression
  • -v "from" - print value of "from" XPath expression
  • sample.xml - test input file used
Do you want to see the xslt used in the background? No problem, just go for:
xmlstarlet sel -C -t -m "messages" -m "note" -v "from" < sample.xml
to see:
<?xml version="1.0"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exslt="http://exslt.org/common" version="1.0" extension-element-prefixes="exslt">
  <xsl:output omit-xml-declaration="yes" indent="no"/>
  <xsl:template match="/">
    <xsl:for-each select="messages">
      <xsl:for-each select="note">
        <xsl:call-template name="value-of-template">
          <xsl:with-param name="select" select="from"/>
        </xsl:call-template>
      </xsl:for-each>
    </xsl:for-each>
  </xsl:template>
  <xsl:template name="value-of-template">
    <xsl:param name="select"/>
    <xsl:value-of select="$select"/>
    <xsl:for-each select="exslt:node-set($select)[position()&gt;1]">
      <xsl:value-of select="'&#10;'"/>
      <xsl:value-of select="."/>
    </xsl:for-each>
  </xsl:template>
</xsl:stylesheet>
easy, right?

Selecting xml element

Do you need whole element, not just value? Go for:
xmlstarlet sel -t -m "messages" -m "note" -c "from" < test.xml
to see:
<from>Jani</from><from>Tove</from>
Where only new flag introduced is:
  • -c "from" - print value of "from" XPath expression

Attribute values

Let's end the examples with id-attribute values. Let's assume, we're interested in the note's id attribute values:
xmlstarlet sel -t -m "messages" -m "note" -v "@id" < test.xml
should result in:
501502

Conclusion

XmlStarlet is much more powerful that shown in the examples above.
However my goal was not to show it all (if interested, go for official docs) but rather to show you the tool that can help with xml processing in the shell scripts.
It can save you from quite some unneeded complexity possibly introduced by sed/awk/grep solutions as it respects commented sections, ... and brings the full XPath power to your scripting.

Well, I need to admit, that the tool's development is somewhat stalled. Usually, that's not a good indicator. Still I consider it mature and extremely useful.