U1 Research ENGLISH http://u1research.org/blogs/en U1 Research you need us Thu, 07 Jul 2011 10:14:36 +0000 en hourly 1 http://wordpress.org/?v=3.1.2 How to Install Oracle 11gR2 on Debian vserver http://u1research.org/blogs/en/2010/05/17/oracle11gr2-debian-vserver/ http://u1research.org/blogs/en/2010/05/17/oracle11gr2-debian-vserver/#comments Mon, 17 May 2010 06:15:34 +0000 admin Good to know]]> http://u1research.org/blogs/en/?p=137 We are going to install an Oracle 11gR2 database onto a Debian VServer (an software environment with own application running space that shares the kernel with host machine). VServer and non-VServer based installation differs only on how to set kernel parameters.

First steps

  • We have installed a simple (no “task” selection) 64 bites Debian Linuxot (5.0 – Lenny) on our VServer, and we become root at the beginning.
  • Add the following users and groups to your system, i.e. run the following commands:
    adduser –group dba
    adduser –group oinstall
    adduser –gid 65534 –group nobody
    useradd -g oinstall -G dba -p put_your_password_here oracle
  • Since the whole operating system is a guest host we will do the maintainance from remote console. Remember to set the proper port forwarding data (SSH, Oracle Listener) on host server. In order to access the server, and for Oracle installation we set up the following Debian packages:
    apt-get install libedit2 libgpm2 libkeyutils1 libkrb53 libx11-6 libx11-data libxau6 \
    libxcb-xlib0 libxcb1 libxdmcp6 libxext6 libxmuu1 openssh-blacklist openssh-blacklist-extra \
    openssh-client openssh-server vim vim-runtime x11-common x11-utils xauth xterm \
    less unzip make binutils gcc libaio-dev libc6-dev ia32-libs rpm libc6-dev-i386 unixODBC-dev \
    pdksh expat sysstat elfutils libebl-dev gawk libstdc++5
  • The next is to make Debian RedHat “compatible”:
    ln -s /usr/bin/awk /bin/awk
    ln -s /usr/bin/rpm /bin/rpm
    ln -s /usr/bin/basename /bin/basename # Suggested by Giuseppe Sacco
    ln -s /etc /etc/rc.d # Required for root.sh
  • We should modify kernel settings on the host machine. To be more precise modify /etc/sysctl.conf file on the host server according to Oracle 11gR2 needs:
    kernel.shmmax = 536870912
    kernel.sem = 250 32000 100 128
    fs.file-max = 6815744
    net.core.rmem_default = 262144
    net.core.rmem_max = 4194304
    net.core.wmem_max = 1048576
    net.core.wmem_default = 262144
    fs.aio-max-nr = 1048576
    net.ipv4.ip_local_port_range = 9000 65500
  • Now run the following command as root (on the host machine, of course):
    sysctl -p
  • [Eventual problem] A problem might arise if VServer /tmp partition is too small then some application (e.g. lynx) like to store temporary files there fail. If file downloads is not a case then you can skip this step.
  • We download Oracle Database 11g R2 install packages including the first, and the second CD.

Installation

  • In order to run installation file at the first time on Debian which is not a supported platform – run the following commands:
    su – oracle
    export ORACLE_HOME=/where/you/will/install/your/files
    export DISPLAY=IP.of.your.display.machine.com:0.0
    cd /where/you/will/put/your/install/files
    unzip /download/location/linux.x64_11gR2_database_1of2.zip
    unzip /download/location/linux.x64_11gR2_database_2of2.zip
    cd database/
    ./runInstaller -ignoreSysPrereqs
  • Graphic interface is now started, and it will ask the following questions:
    • Partioning option: it enables to store table in small data files called partitions – it is very effective for large databases.
    • OLAP option: a new option which supports efficient pivoting and OLAP cube manipulations.
    • Label Security option: supports limitations of user accesses to a single record or fields within records.
    • Oracle Data Mining option: very good in-database mining support (the best but expensive solution)
    • Database Vault option: enables full scale auditing on databases (you need this for Audio Vault), or limiting administrator role users (it is important if sensitive data are stored).
    • Real Application Testing: a testing environment with versioning support to see how your software will work in (the next) changed database environment.

    • 1. If a critical error arise or an update is available then we either ask database to email us or we can use our Metalink account to get information. Since we are making a simple installation we Skip this step.

    2. We choose to whether to set up our database in “playground” mode or in “server” mode. We have chosen Server Mode hence we can set up more options.

    3. The next question is whether we want to use a single database or a database cloud called grids. Single instance.

    4. Are you looking for an interactionless or a professional mode? The former one is an easy way so we choose the latter one. Advanced mode enables more detailed server configuration.

    5. Database native language support is to be set. It sets keyboard settings to default, nevertheless English is very very recommended to be selected in either cases. We have English and Hungarian (our native language) on the right side.

    6. The next tab asks for new options. We can choose between Enterprise Edition (EE), Standard Edition (SE), or Standard Edition One (SE1). This is a good news (and new feature) since in the past, there were 3 different setup kits for these very similar options. If we are looking for database options then EE is the only way. If you are looking for a relatively cheap database (no option) license then SE is enough. If we are a small company with some database served users, and we are not supposed to grow fast in the next 3-5 years then choose SE1. Note that, there exists a free Oracle database license called XE with some limitations (1 core, up to 4GB disc usage, single instance, no grid). We are looking for database options so we choose EE. But… Remember that the most important question is the “Check Options…” button on this page. You can not change these settings later easily, and more importantly it is illegal to install database options without appropriate license (with the exception of non-profit research, teaching – free academy license is required -, and other development – free developer license is required):

    7. Now, it is asked where to put Oracle database management and data files (ORACLE_HOME). If this shell environment is not set do not panic! They are asking now. We suggest that you replace default “dbhome_1″ by something more meaningful name.The next page wants you to declare where to put Oracle common working space (Inventory) used by and required for all(!) Oracle products. That is why you should place these files in a dba group or a commonly writeable, and at least dba group readable directory.

    8. You can choose between transactional (OLTP – many data insert, small queries) and analytical (OLAP – batch insert, reporting, special queries) use of your database. If you need to install a data warehouse then you shall choose OLAP. If you know almost nothing about these notions then OLTP is your choice.

    9. You can set database “global name” which is assumed to be a unique identifier. According to conventions it is a concatenation of SID (local system identifier of your database) and your DNS (server name). If the first word is replaced then it will appear as your SID. Watch out! Non-English characters should not be used here because it will hurt you very very very deep.

    10. In order to confuse you the next window consists of four panels (not 1!). OK, let’s see:

     

  • You can set the global memory space called SGA (Shared Global Area) which is “brain”, cache, operating memory, short and long term memory as well. Large is good for Oracle database management, and bad for all other application. Be wise! If database usage is rather a background application then it might not be a bottleneck so it is wise to set it relatively low (but >1GB). If database intensive processes are common then set SGA to be large but less than 80% (give memory to other applications like operating system).
  • Default character set (UTF-32) is usually the perfect choice. If you do not know what exactly you are doing then do not touch it.
  • Usual paranoid settings can be adjusted here. Paranoia in this case means that you can set how complex (hard to guess) be a password, how often users should change, how many times it can be used in a row, how long a user is banned, etc. Only sadists, unconfidents on colleagues, and those who want to protect stored data according to law (e.g. sensitive data) turn this on.
  • Finally, we can choose whether to install example (scott/tiger) database or not. It is very handy when you are learning, or trying to understand manual. Otherwise it is only a waste of space.11. If you like to get email notification on system errors then set here who, how, and when to get emails.12. Do you prefer file system or ASM (Automatic Storage Management) based storage? If database is in limited used, and you do not intent to store large binary or text files then file system rules. Compression seems to be hardly useful but they say it makes database management fast (our tests indicate only 2-3% gain). We have preferred file system based storage.

    13. Do you need automated data backups? Choose No unless it is mission critical.
    14. You can set passwords for the most “powerful” database users – we have set all at once. Do not forget your password (and noow you face your previous paranoia settings).
    OSDBA, OSOPER.

    15. On this screen great many Oracle verified errors are put. The thing is setup checks the settings with rpm -q command – and no wonder – it won’t work well for Debian (and packages). Because of using VServer some kernel settings also would fail. But, you have set kernel settings on the host machine, don’t you??? Ignore all.
    16. At this point setup smoothly filled the progress bar from 0 to 100%, including database creation. At last it throws a window which tells you run the following commands as root:

    /where/your/oraInventory/is/located/orainstRoot.sh # oraInventory’s location, see step #8
    /where/your/database/binaries/are/located/root.sh # ORACLE_HOME, see step #7
]]> http://u1research.org/blogs/en/2010/05/17/oracle11gr2-debian-vserver/feed/ 0 On the precision rate of data mining models… http://u1research.org/blogs/en/2010/05/05/precision-of-a-model/ http://u1research.org/blogs/en/2010/05/05/precision-of-a-model/#comments Wed, 05 May 2010 06:00:53 +0000 Data Mining]]> http://u1research.org/blogs/en/?p=128 There is no question I hate more than ‘What is the overall precision (or goodness) of your data mining model?’. The most of the people I have met tend to think that a model far from 100% good is worth nothing, or at least it is not ready for the market. In the other hand, people often think that a model close to 100% is very good and needs no improvement at all. Nothing else but this can be very misleading. Precision as a unique and a common measure for data mining models leads to really bad strategies; a common mistake committed even by qualified engineers and managers. In order to avoid the bad interpretation of my answer I like to tell how our data mining model differs from others with pros and cons including precision rates. It sometimes seems that I avoid the question so now I try to answer the question as completely as possible, and I show my points worth to analyze before any decisions are made.

First of all, it is necessary to define what does ‘presicion’ means. While great many definition is known from the literature, commonly one states that precision is the overall (or average) rate of good guesses or decisions in data sets. That is, if we have 72 exact guesses and 11 false tips then the precision rate is 72 / ( 72 + 11 ) = 86.75%.

Besides the good guesses one should take into account the following data mining properties hence they are very closely related to precision rates, and sometimes they are more important or even crucial to meet at least a certain measure.

  • General precision. If we have a very promising algorithm it is worth to test it in international grounds. It is good to know that every task and every real life situation are different, there is no algorithm fit for any problem (unless you are writing a dissertation). Make a deep analysis why your algorithm adopts well or wrong to a problem, and hey, never give up! There is no reason to think that the best known algorithm for some problem is the best for the problem were are working on now. In 2007, an international challenge were held for analyzing generalization behaviors of algorithms. Results were no surprising there were found no universal solution even tricky combinations of algorithms are failed in some problems. Do you have ever met or written a marketing folio telling that product embodied solutions are far more better in general than those of competitors? Did you ever believe / prove that?
  • Relative, data dependent precision. In my point of view, determining baseline precision rate is the most important task of a data mining project. To tell you the truth I rather prefer this value than precision rate. It is worth to set baseline to the best known previous precision rate if it is available. In general, whenever customers need enhancements on a previously made model it clearly shows that they deeply understand how e.g. 1% improvement in precision boosts their business. Moreover, they usually know how to marketing the results.
  • Simplicity. Precision rate of a data mining model depends on the number of input parameters, i.e. on the information base it is working with. It is analogous to Occam’s razor problem: we prefer solutions (businesses, people) end-up with the same result (profit, information) from less information (money, communication). That kind of ‘intellectual highness’ can be converted easily to money hence a more simple model is faster, needs less resources and workarounds. While it seems natural to make a shot on this I have never met customers asking me about operational costs of a model. It happened in a case that a model with the best known precision rate required more money than model should spare for a company.
  • Precision in time. Many solutions working with the ‘more information more precise model’ motto is based on a false logic, or at least their precision rates degrade fast. Let me explain this with an example. Imagine that we have 8 apples with 2 distinctive properties, e.g. volume and color, and our task is to form 4 groups of them. It is easy to see that partitioning different apples based on key properties is not a hard task. In fact, if each apple belongs to exactly one group of four one can find a proper partitioning algorithm which results the same groups. In the case of machine learning and mining models ‘distinctive’ usually means linear independency. In general, it is true that if have M distinctive properties, T unique items, and M / T is less or equal to 2 than the problem can be learned ideally just like in the case of apples. Now, one can identify that number of records in an international data set are not changing over time, so M is constant. That is, precision can be easily improved by increasing T. At a cost of what? Well, we can have the most impressive precision rate, we can publish it anywhere, or we can get a Ph.D. for that. Nevertheless, the number of data points in real life is not limited so there will come the hour of truth as M / T is getting far more than 2. Same happens when students learn reference books by heart for exams. Knowledge they have seem thrilling until the graduation and (obviously) useless at work. In the other hand, wise data miners should know they learn the problem alongside with data mining models and this human learning must not built-in during the process. In this apple case, we made a hint for data mining model about the number of data points, i.e. we have just cheated the problem.
  • On the information relevancies. Usually more information available leads to a more precise model assuming information have similar relevancies or qualities. What happens if more information means more indirectly related sources are used? Algorithms treat information equally no data mining algorithm supports prejudice or racism, and does not distinguish soil from diamond. In other words: garbage in garbage out. If information quality differs significantly then more non-relevant information increase noise in data, and consequently decrease precision rate. At this point you may say: “Hey, that is why data cleaning and preprocessing is so important, what is the point here?” In case of preprocessing data is transformed from initial space S1 into another SA in order to find correlations between data. However, when data mining models are built S1 space is transformed into SM, and at the most of the cases SM and SA are not the same. Therefore preprocessing rather should be done at SM space which is not the case in data mining or else it may not calculate proper relevancy for data. Good data mining models pay attention to information quality, they discover important information elements and/or latent correlations between them. In some cases, e.g. in cancer diagnostics, it is rather important to find signs and reasons for cancers than to build a more precise model.
  • Critical errors, sequences. A good face recognition solution has cca. 97-98% precision rate which does not find terrorists actually but it is able to recognize employees with no twins. In reality no image based face recognition algorithms are that good, so don’t worry if you are not there. If you have a close to 85% algorithm, and you know that 2-3 seconds videos contain lots (12-75) of somewhat different (cca. 8-10 independent) face images then many attempts for recognition converge into a specific result. It is easy to prove that if error rate depends on video quality nuances then this 85% algorithm can be transformed into 97% recognition rate solution (see e.g. voting strategies). That is, precision rate can be extended easily if you know more about the usage.
  • Behind the curtains. The most common passenger counters also provide 97-98% precision rates. Datasheets do not remember how this rate should be treated. Note that it is hard to tell how to calculate precision in a public transportation vehicle: is error rate calculated after every stops or is it weighted on uniform time periods? A good performance on the former one implies that algorithm is possibly more precise on rush hours hence nearly empty vehicles generate lower error rate than vehicles with many people. A good performance on the latter one indicates the opposite since rush hours form a relatively small portion of the day. In either cases error rate may have a large deviation, i.e. sometimes it might reach even 10%. Assume that precision rate is calculated on a daily basis by comparing results to manual counts then a solution optimized for this rate focuses on general attributes of customers, e.g. their heights, weights. That is, if it works for Europe it should not be sold to China.
  • When 99% is still bad. I read some times ago a scientific like leaflet on a “magic gadget” which promises to tell me whether I have a specific disease or not. It stated that e.g. meningitis is recognized by 99,2% (with no guarantee). Wow! The only tiny little problem is to have a meningitis has 5 from 1,000,000 apriori chance. At this point I have suddenly realized that I can produce in no time a much more precise “machine” with 99,9999% precision rate. My solution would repeat that “you have no meningitis”. I believe in probabilities. Nevertheless, I think we should rather go to a doctor.
  • Precision and quantities. Good optical character recognizers (OCR) have 98-99% precision rate for English. I have often heard from members of SMEs that electronic document digitalization is cheap, simple, and 1-2% is low enough not to worry about. But what does 2% means in this case? It means on an average basis there are 2 non-recognized from 100 letters. Since an English sentence consists of 7-10 words, and a word consists of 5-7 letters in general it means there is 1 error per sentence as an average. As a result we get a digitalized but poor quality text.
  • When 55% is very good. Predicting the future is a hard task even for stock exchange problems hence data elements are constantly changing their relative importance. From an observer point of view a stock exchange problem is rather a random or stochastic process (like gambling). For example, to guess whether tomorrow the closing price for Google will be higher or lower than it was today is very similar to spin a coin. In order to build a winning strategy we should change uniform apriori probabilities of different sides. Assume that we have 1% charges for transactions there can be easily found a winning strategy based on a 55% predictor, and with enough amount of money. You just need to guarantee that average amount of losses is compensated by average amount of gains (i.e. use stop loss).

    • As a consequence, I would say that “What is the precision of your data mining model?” and “How good is you model?” are ill-stated questions. Simple answers for these questions must be misleading at some point or they could feed illusions about a certain application. Nevertheless, precision rate is important we just need a more general (or complex) measure which helps stakeholders to properly understand the risks they take by using a data mining model.

    ]]>     http://u1research.org/blogs/en/2010/05/05/precision-of-a-model/feed/ 0     KDDCup 2010 http://u1research.org/blogs/en/2010/04/19/kddcup-2010/ http://u1research.org/blogs/en/2010/04/19/kddcup-2010/#comments Mon, 19 Apr 2010 20:00:29 +0000 U1 Research]]> https://www.u1research.org/blog/?p=70 ACM SIGMOD KDDCup is always a big challenge hence it is the biggest data mining competition in the world. Last 5 years 200+ research teams had sumbitted results from 30+ countries. These year, competitors can help students to improve their skills by leading them through e-learning material. While the problem is new in KDDCup series it seems that collaborative filtering still rules. For more details, see the official page.

    U1 Research always pays special attention to the ACM KDDCup series. Members of U1 Research first have participated at KDDCups in 2005, and they were awarded runner-up. Later on, we also attended the challenge in 2006 (we had got winner and a runner-up award), and in 2008 (6th prize award – the best European team). Our company founders formed the successful 2008 team. In 2009, we started together a new company so we had no time for the competition. It seems something is not changing… next time, maybe.

    ]]>     http://u1research.org/blogs/en/2010/04/19/kddcup-2010/feed/ 0     U1 Research has joined to MMKlaszter http://u1research.org/blogs/en/2010/04/08/joined-to-mmklaszter/ http://u1research.org/blogs/en/2010/04/08/joined-to-mmklaszter/#comments Thu, 08 Apr 2010 10:00:48 +0000 U1 Research]]> https://www.u1research.org/blog/?p=62 Clusters provide a new form of cooperation here in Hungary. The Mobility and Multimedia Cluster (MMKlaszter) was founded two years ago by local branches of multi-national companies like Magyar Telekom (a Deutsche Telekom subsidiary), Hewlett-Packard, Ericsson, Microsoft, the top Hungarian universities, and the most important Hungarian IT SMEs. MMKlaszter have grown quickly, it is now the most important cluster in Hungary. MMKlaszter effectively combines IT forces, and it currently drives European market entrances of cutting edge technologies and innovative companies.

    Members of U1 Research have taken part in MMKlaszter’s life from the beginnings, including legislative and funds related issues. We have participated in MMKlaszter’s innovation challenge in 2009 with great success. We strongly believe in that U1 Research revolutionary IT solutions with the help of MMKlaszter can pave the way for other Hungarian innovative companies to enter European markets.

    ]]>     http://u1research.org/blogs/en/2010/04/08/joined-to-mmklaszter/feed/ 0     U-8 National Chess Tournament http://u1research.org/blogs/en/2010/03/14/u8-chess/ http://u1research.org/blogs/en/2010/03/14/u8-chess/#comments Sun, 14 Mar 2010 12:00:28 +0000 U1 Research]]> https://www.u1research.org/blog/?p=80 U-8 National Chess Tournament was held by the Hungarian Chess Federation between 12 and 14 March in Budapest at Losonci Primary School. The tournament was sponsored by the Losonci Foundation. Since the most of the 8-year-old participants is quite young for writing U1 Research was willing to record and to share the top boards’ games. As a result, U1 Research provided up-to-date free boulletins between rounds. The experiment was a great success among players and coaches as well. We are proud to be helpful at the first Hungarian U-8 tournament with boulletins.

    ]]>     http://u1research.org/blogs/en/2010/03/14/u8-chess/feed/ 0