Changeset 4108


Ignore:
Timestamp:
2010-06-29 13:08:25 (3 years ago)
Author:
bruno
Message:

Make it possible to run tests against an external HBase of your choice.
Update README.txt accordingly (+ reflow at 80 columns).
HBaseProxy: delete the dummy row that is inserted to detect if the compaction has run.

Location:
projects/lily/trunk
Files:
4 edited

Legend:

Unmodified
Added
Removed

  • projects/lily/trunk/README.txt

    r4107 r4108  
    5252=============== 
    5353 
    54 Log output of testcases is by default sent to a target/log.txt, errors are however always logged to the 
    55 console. Debug output to the console of selected log categories can be enabled by running tests as follows: 
     54Log output of testcases is by default sent to a target/log.txt, errors 
     55are however always logged to the console. Debug output to the console of 
     56selected log categories can be enabled by running tests as follows: 
    5657 
    5758mvn -Plog test 
    … …  
    5960This is mostly useful when working on individual subprojects/tests. 
    6061 
    61 Running tests faster 
    62 -------------------- 
     62Running tests (faster) against an existing HBase 
     63------------------------------------------------ 
    6364 
    64 Test run rather slow because HBase-based tests launch a mini Hadoop/ZooKeeper/HBase-cluster as part of the 
    65 testcase. While this takes some time in itself, it is especially the creation of tables in HBase which 
     65Test run rather slow because HBase-based tests launch a mini 
     66Hadoop/ZooKeeper/HBase-cluster as part of the testcase. While this takes 
     67some time in itself, it is especially the creation of tables in HBase which 
    6668takes time. 
    6769 
    68 The tests can be sped up by starting an independent cluster and running the tests against that. Instead of 
    69 dropping and recreating tables between each test, the tables are emptied by deleting all rows from them (thus 
    70 be very careful against which HBase you run this!). 
     70The tests can be sped up by starting an independent cluster and running the 
     71tests against that. Instead of dropping and recreating tables between each 
     72test, the tables are emptied by deleting all rows from them (thus be very 
     73careful against which HBase you run this!). Besides the speed advantages, 
     74this is also easier for debugging. 
    7175 
    72 The easy way to do this is: 
     76 
     77                     === Quick way: dummy HBase === 
     78 
     79 
     80To easily launch a mini HBase without having to install it, execute: 
    7381 
    7482cd testfw 
    … …  
    7684 
    7785This prints a line "Minicluster is up" when it is started, though it is 
    78 quickly followed by more logging so you might not easily notice it. 
     86quickly followed by more logging. 
    7987 
    8088And then run the tests with 
    … …  
    8290mvn -Pconnect test 
    8391 
    84 The first time this will still take more time (though already quite a bit less than before), since the 
    85 tables still need to be created. Subsequent runs should be way faster. 
     92The first time this will still take more time (though already quite a bit 
     93less than before), since the tables still need to be created. Subsequent 
     94runs should be way faster. 
    8695 
    87 Note that profiles can be combined, for example: 
     96Each time this 'mini' HBase is restarted, it looses its state so the first run 
     97after restart will again be a bit slower. 
     98 
     99When you run testcases without -Pconnect, and you have a mini HBase launched, 
     100it might get confused (because some of the same ports are used), and you 
     101might need to restart it. 
     102 
     103 
     104                 === Run against existing cluster === 
     105 
     106 
     107If you want to connect to an HBase you have installed, you need to specify 
     108the name(s) and port of Zookeeper: 
     109 
     110mvn -Pconnect -DargLine="-Dlily.test.hbase.zookeeper.quorum=localhost -Dlily.test.hbase.zookeeper.property.clientPort=2181" test 
     111 
     112More in general, any HBase property can be specified by prefixing it with 
     113"lily.test." (also when the tests run with an embedded HBase). 
     114 
     115Combining profiles 
     116================== 
     117 
     118Maven profiles can be combined, for example: 
    88119 
    89120mvn -Pconnect -Plog test 

  • projects/lily/trunk/pom.xml

    r4107 r4108  
    315315        <version>2.5</version> 
    316316          <configuration> 
    317             <argLine>-Xmx256m</argLine> 
     317            <argLine>-Xmx256m ${argLine}</argLine> 
    318318            <systemPropertyVariables> 
    319319              <lily.test.hbase>${lily.test.hbase}</lily.test.hbase> 

  • projects/lily/trunk/testfw/src/main/java/org/lilycms/testfw/HBaseProxy.java

    r4102 r4108  
    6161        System.out.println("HBase usage mode: " + MODE); 
    6262 
     63        CONF = HBaseConfiguration.create(); 
     64 
    6365        switch (MODE) { 
    6466            case EMBED: 
    65                 // TODO use optimized hbase config 
    66                 TEST_UTIL = new HBaseTestingUtility(); 
     67                addHBaseTestProps(CONF); 
     68                addUserProps(CONF); 
     69                TEST_UTIL = new HBaseTestingUtility(CONF); 
    6770                TEST_UTIL.startMiniCluster(1); 
    6871                CONF = TEST_UTIL.getConfiguration(); 
    6972                break; 
    7073            case CONNECT: 
    71                 CONF = HBaseConfiguration.create(); 
    7274                CONF.set("hbase.zookeeper.quorum", "localhost"); 
    7375                CONF.set("hbase.zookeeper.property.clientPort", "21812"); // matches HBaseRunner 
     76                addUserProps(CONF); 
    7477                cleanTables(); 
    7578                break; 
    … …  
    7780                throw new RuntimeException("Unexpected mode: " + MODE); 
    7881        } 
     82    } 
     83 
     84    /** 
     85     * Adds all system property prefixed with "lily.test.hbase." to the HBase configuration. 
     86     */ 
     87    private void addUserProps(Configuration conf) { 
     88        Properties sysProps = System.getProperties(); 
     89        for (Map.Entry<Object, Object> entry : sysProps.entrySet()) { 
     90            String name = entry.getKey().toString(); 
     91            if (name.startsWith("lily.test.hbase.")) { 
     92                String hbasePropName = name.substring("lily.test.".length()); 
     93                conf.set(hbasePropName, entry.getValue().toString()); 
     94            } 
     95        } 
     96    } 
     97 
     98    protected static void addHBaseTestProps(Configuration conf) { 
     99        // The following properties are from HBase's src/test/resources/hbase-site.xml 
     100        conf.set("hbase.regionserver.msginterval", "1000"); 
     101        conf.set("hbase.client.pause", "5000"); 
     102        conf.set("hbase.client.retries.number", "4"); 
     103        conf.set("hbase.master.meta.thread.rescanfrequency", "10000"); 
     104        conf.set("hbase.server.thread.wakefrequency", "1000"); 
     105        conf.set("hbase.regionserver.handler.count", "5"); 
     106        conf.set("hbase.master.info.port", "-1"); 
     107        conf.set("hbase.regionserver.info.port", "-1"); 
     108        conf.set("hbase.regionserver.info.port.auto", "true"); 
     109        conf.set("hbase.master.lease.thread.wakefrequency", "3000"); 
     110        conf.set("hbase.regionserver.optionalcacheflushinterval", "1000"); 
     111        conf.set("hbase.regionserver.safemode", "false"); 
    79112    } 
    80113 
    … …  
    181214            HTable htable = new HTable(CONF, tableName); 
    182215 
     216            byte[] tmpRowKey = Bytes.toBytes("HBaseProxyDummyRow"); 
     217            byte[] CF = EXPLOIT_TIMESTAMP_TABLES.get(tableName); 
     218            byte[] COL = Bytes.toBytes("DummyColumn"); 
     219 
    183220            byte[] value = null; 
    184221            while (value == null) { 
    185                 byte[] tmpRowKey = Bytes.toBytes("HBaseProxyDummyRow"); 
    186                 byte[] CF = EXPLOIT_TIMESTAMP_TABLES.get(tableName); 
    187                 byte[] COL = Bytes.toBytes("DummyColumn"); 
    188222                Put put = new Put(tmpRowKey); 
    189223                put.add(CF, COL, 1, new byte[] { 0 }); 
    … …  
    199233                Thread.sleep(100); 
    200234            } 
     235 
     236            // Delete our dummy row again 
     237            htable.delete(new Delete(tmpRowKey)); 
    201238        } 
    202239    } 

  • projects/lily/trunk/testfw/src/main/java/org/lilycms/testfw/HBaseRunner.java

    r4090 r4108  
    3939        conf = HBaseConfiguration.create(); 
    4040 
    41         // The following properties are from HBase's src/test/resources/hbase-site.xml 
    42         conf.set("hbase.regionserver.msginterval", "1000"); 
    43         conf.set("hbase.client.pause", "5000"); 
    44         conf.set("hbase.client.retries.number", "4"); 
    45         conf.set("hbase.master.meta.thread.rescanfrequency", "10000"); 
    46         conf.set("hbase.server.thread.wakefrequency", "1000"); 
    47         conf.set("hbase.regionserver.handler.count", "5"); 
    48         conf.set("hbase.master.info.port", "-1"); 
    49         conf.set("hbase.regionserver.info.port", "-1"); 
    50         conf.set("hbase.regionserver.info.port.auto", "true"); 
    51         conf.set("hbase.master.lease.thread.wakefrequency", "3000"); 
    52         conf.set("hbase.regionserver.optionalcacheflushinterval", "1000"); 
    53         conf.set("hbase.regionserver.safemode", "false"); 
     41        HBaseProxy.addHBaseTestProps(conf); 
    5442 
    5543        startMiniCluster(1); 
Note: See TracChangeset for help on using the changeset viewer.