forked from apache/pig
-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathCHANGES.txt
3570 lines (1950 loc) · 127 KB
/
CHANGES.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
Pig Change Log
Trunk (unreleased changes)
INCOMPATIBLE CHANGES
PIG-3191: [piggybank] MultiStorage output filenames are not sortable (Danny Antonelli via jcoveney)
PIG-3174: Remove rpm and deb artifacts from build.xml (gates)
IMPROVEMENTS
PIG-3202: CUBE operator not documented in user docs (prasanth_j via billgraham)
PIG-3189: Remove ivy/pig.pom and improve build mvn targets (billgraham)
PIG-3192: Better call to action to download Pig in docs (rjurney via jcoveney)
PIG-3167: Job stats are printed incorrectly for map-only jobs (Mark Wagner via jcoveney)
PIG-3169: Remove temporary files that are not needed (Mark Wagner via jcoveney)
PIG-3131: Document PluckTuple UDF (rjurney via jcoveney)
PIG-3098: Add another test for the self join case (jcoveney)
PIG-3129: Document syntax to refer to previous relation (rjurney via jcoveney)
PIG-2553: Pig shouldn't allow attempts to write multiple relations into same directory (prkommireddi via cheolsoo)
PIG-3179: Task Information Header only prints out the first split for each task (knoguchi via rohini)
PIG-3108: HBaseStorage returns empty maps when mixing wildcard with other columns (christoph.bauer via billgraham)
PIG-3178: Print a stacktrace when ExecutableManager hits an OOM (knoguchi via rohini)
PIG-3160: GFCross uses unnecessary loop (sandyr via cheolsoo)
PIG-3138: Decouple PigServer.executeBatch() from compilation of batch (pkommireddi via cheolsoo)
PIG-2878: Pig current releases lack a UDF equalIgnoreCase.This function returns a Boolean value indicating whether string left is equal to string right. This
check is case insensitive. (shami via gates)
PIG-2994: Grunt shortcuts (prasanth_j via cheolsoo)
PIG-3140: Document PigProgressNotificationListener configs (billgraham)
PIG-3139: Document reducer estimation (billgraham)
PIG-2764: Add a biginteger and bigdecimal type to pig (jcoveney)
PIG-3073: POUserFunc creating log spam for large scripts (jcoveney)
PIG-3082: outputSchema of a UDF allows two usages when describing a Tuple schema (jcoveney)
PIG-3124: Push FLATTENs After FILTERs If Possible (nwhite via daijy)
PIG-3086: Allow A Prefix To Be Added To URIs In PigUnit Tests (nwhite via gates)
PIG-3091: Make schema, header and stats file configurable in JsonMetadata (pkommireddi via jcoveney)
PIG-3078: Make a UDF that, given a string, returns just the columns prefixed by that string (jcoveney)
PIG-3090: Introduce a syntax to be able to easily refer to the previously defined relation (jcoveney)
PIG-3057: Make PigStorage.readField() protected (pablomar and billgraham via billgraham)
PIG-2788: improved string interpolation of variables (jcoveney)
PIG-2362: Rework Ant build.xml to use macrodef instead of antcall (azaroth via cheolsoo)
PIG-2857: Add a -tagPath option to PigStorage (prkommireddi via cheolsoo)
PIG-2341: Need better documentation on Pig/HBase integration (jthakrar and billgraham via billgraham)
PIG-3075: Allow AvroStorage STORE Operations To Use Schema Specified By URI (nwhite via cheolsoo)
PIG-3062: Change HBaseStorage to permit overriding pushProjection (billgraham)
PIG-3016: Modernize more tests (jcoveney via cheolsoo)
PIG-2582: Store size in bytes (not mbytes) in ResourceStatistics (prkommireddi via billgraham)
PIG-3006: Modernize a chunk of the tests (jcoveney via cheolsoo)
PIG-2997: Provide a convenience constructor on PigServer that accepts Configuration (prkommireddi via rohini)
PIG-2933: HBaseStorage is using setScannerCaching which is deprecated (prkommireddi via rohini)
PIG-2881: Add SUBTRACT eval function (jocosti via cheolsoo)
PIG-3004: Improve exceptions messages when a RuntimeException is raised in Physical Operators (julien)
PIG-2990: the -secretDebugCmd shouldn't be a secret and should just be...a command (jcoveney)
PIG-2941: Ivy resolvers in pig don't have consistent chaining and don't have a kitchen sink option for novices (jgordon via azaroth)
PIG-2778: Add 'matches' operator to predicate pushdown (cheolsoo via jcoveney)
PIG-2966: Test failures on CentOS 6 because MALLOC_ARENA_MAX is not set (cheolsoo via sms)
PIG-2794: Pig test: add utils to simplify testing on Windows (jgordon via gates)
PIG-2910: Add function to read schema from outout of Schema.toString() (initialcontext via thejas)
OPTIMIZATIONS
PIG-3013: BinInterSedes improve chararray sort performance (rohini)
BUG FIXES
PIG-3135: HExecutionEngine should look for resources in user passed Properties (prkommireddi via cheolsoo)
PIG-3200: MiniCluster should delete hadoop-site.xml on shutDown (prkommireddi via cheolsoo)
PIG-3158: Errors in the document "Control Structures" (miyakawataku via cheolsoo)
PIG-3161: Update reserved keywords in Pig docs (russell.jurney via cheolsoo)
PIG-3156: TestSchemaTuple fails in trunk (cheolsoo)
PIG-3155: TestTypeCheckingValidatorNewLP.testSortWithInnerPlan3 fails in trunk (cheolsoo)
PIG-3154: TestPackage.testOperator fails in trunk (dreambird via cheolsoo)
PIG-3168: TestMultiQueryBasic.testMultiQueryWithSplitInMapAndMultiMerge fails in trunk (cheolsoo)
PIG-3137: Fix Piggybank test to not using /tmp dir (dreambird via cheolsoo)
PIG-3149: e2e build.xml still refers to jython 2.5.0 jar even though it's replaced by jython standalone 2.5.2 jar (cheolsoo)
PIG-2266: Bug with input file joining optimization in Pig (jadler via cheolsoo)
PIG-2645: PigSplit does not handle the case where SerializationFactory returns null (shami via gates)
PIG-3031: Update Pig to use a newer version of joda-time (zjshen via cheolsoo)
PIG-3071: Update hcatalog jar and path to hbase storage handler jar in pig script (arpitgupta via cheolsoo)
PIG-3029 TestTypeCheckingValidatorNewLP has some path reference issues for cross-platform execution (jgordon via gates)
PIG-3120: setStoreFuncUDFContextSignature called with null signature (jdler via cheolsoo)
PIG-3115: Distinct Build-in Function Doesn't Handle Null Bags (njw45 via daijy)
PIG-2433: Jython import module not working if module path is in classpath (rohini)
PIG-2769 a simple logic causes very long compiling time on pig 0.10.0 (nwhite via gates)
PIG-2251: PIG leaks Zookeeper connections when using HBaseStorage (jamarkha via cheolsoo)
PIG-3112: Errors and lacks in document "User Defined Functions" (miyakawataku via cheolsoo)
PIG-3050: Fix FindBugs multithreading warnings (cheolsoo)
PIG-3066: Fix TestPigRunner in trunk (cheolsoo)
PIG-3101: Increase io.sort.mb in YARN MiniCluste (cheolsoo)
PIG-3100: If a .pig_schema file is present, can get an index out of bounds error (jcoveney)
PIG-3096: Make PigUnit thread safe (cheolsoo)
PIG-3095: "which" is called many, many times for each Pig STREAM statement (nwhite via cheolsoo)
PIG-3085: Errors and lacks in document "Built In Functions" (miyakawataku via cheolsoo)
PIG-3084: Improve exceptions messages in POPackage (julien)
PIG-3072: Pig job reporting negative progress (knoguchi via rohini)
PIG-3014: CurrentTime() UDF has undesirable characteristics (jcoveney via cheolsoo)
PIG-2924: PigStats should not be assuming all Storage classes to be file-based storage (cheolsoo)
PIG-3046: An empty file name in -Dpig.additional.jars throws an error (prkommireddi via cheolsoo)
PIG-2989: Illustrate for Rank Operator (xalan via gates)
PIG-2885: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3 (cheolsoo via sms)
PIG-2928: Fix e2e test failures in trunk: FilterBoolean_23/24 (cheolsoo via dvryaboy)
Release 0.11.0 (unreleased)
INCOMPATIBLE CHANGES
PIG-3034: Remove Penny code from Pig repository (gates via cheolsoo)
PIG-2931: $ signs in the replacement string make parameter substitution fail (cheolsoo via jcoveney)
PIG-1891 Enable StoreFunc to make intelligent decision based on job success or failure (initialcontext via gates)
IMPROVEMENTS
PIG-3044: Trigger POPartialAgg compaction under GC pressure (dvryaboy)
PIG-2907: Publish pig jars for Hadoop2/23 to maven (rohini)
PIG-2934: HBaseStorage filter optimizations (billgraham)
PIG-2980: documentation for DateTime datatype (zjshen via thejas)
PIG-2982: add unit tests for DateTime type that test setting timezone (zjshen via thejas)
PIG-2937: generated field in nested foreach does not inherit the variable name as the field name (jcoveney)
PIG-3019: Need a target in build.xml for source releases (gates)
PIG-2832: org.apache.pig.pigunit.pig.PigServer does not initialize udf.import.list of PigContext (prkommireddi via rohini)
PIG-2898: Parallel execution of e2e tests (iveselovsky via rohini)
PIG-2913: org.apache.pig.test.TestPigServerWithMacros fails sometimes because it picks up previous minicluster configuration file (cheolsoo via julien)
PIG-2976: Reduce HBaseStorage logging (billgraham)
PIG-2947: Documentation for Rank operator (xalan via azaroth)
PIG-2943: DevTests, Refactor Windows checks to use new Util.WINDOWS method for code health (jgordon via dvryaboy)
PIG-2908: Fix unit tests to work with jdk7 (rohini via dvryaboy)
PIG-2965: RANDOM should allow seed initialization for ease of testing (jcoveney)
PIG-2964: Add helper method getJobList() to PigStats.JobGraph. Extend visibility of couple methods on same class (prkommireddi via billgraham)
PIG-2579: Support for multiple input schemas in AvroStorage (cheolsoo via sms)
PIG-2946: Documentation of "history" and "clear" commands (xalan via azaroth)
PIG-2877: Make SchemaTuple work in foreach (and thus, in loads) (jcoveney)
PIG-2923: Lazily register bags with SpillableMemoryManager (dvryaboy)
PIG-2929: Improve documentation around AVG, CONCAT, MIN, MAX (cheolsoo via billgraham)
PIG-2852: Update documentation regarding parallel local mode execution (cheolsoo via jcoveney)
PIG-2879: Pig current releases lack a UDF startsWith.This UDF tests if a given string starts with the specified prefix. (initialcontext via azaroth)
PIG-2712: Pig does not call OutputCommitter.abortJob() on the underlying OutputFormat (rohini via gates)
PIG-2918: Avoid Spillable bag overhead where possible (dvryaboy)
PIG-2900: Streaming should provide conf settings in the environment (dvryaboy)
PIG-2353: RANK function like in SQL (xalan via azaroth)
PIG-2915: Builtin TOP udf is sensitive to null input bags (hazen via dvryaboy)
PIG-2901: Errors and lacks in document "Pig Latin Basics" (miyakawataku via billgraham)
PIG-2905: Improve documentation around REPLACE (cheolsoo via billgraham)
PIG-2882: Use Deque instead of Stack (mkhadikov via dvryaboy)
PIG-2781: LOSort isEqual method (xalan via dvryaboy)
PIG-2835: Optimizing the convertion from bytes to Integer/Long (jay23jack via dvryaboy)
PIG-2886: Add Scan TimeRange to HBaseStorage (ted.m via dvryaboy)
PIG-2895: jodatime jar missing in pig-withouthadoop.jar (thejas)
PIG-2888: Improve performance of POPartialAgg (dvryaboy)
PIG-2708: split MiniCluster based tests out of org.apache.pig.test.TestInputOutputFileValidator (analog.sony via daijy)
PIG-2890: Revert PIG-2578 (dvryaboy)
PIG-2850: Pig should support loading macro files as resources stored in JAR files (matterhayes via dvryaboy)
PIG-1314: Add DateTime Support to Pig (zjshen via thejas)
PIG-2785: NoClassDefFoundError after upgrading to pig 0.10.0 from 0.9.0 (matterhayes via sms)
PIG-2556: CSVExcelStorage load: quoted field with newline as first character sees newline as record end (tivv via dvryaboy)
PIG-2875: Add recursive record support to AvroStorage (cheolsoo via sms)
PIG-2662: skew join does not honor its config parameters (rajesh.balamohan via thejas)
PIG-2871: Refactor signature for PigReducerEstimator (billgraham)
PIG-2851: Add flag to ant to run tests with a debugger port (billgraham)
PIG-2862: Hardcode certain tuple lengths into the TUPLE BinInterSedes byte identifier (jcoveney)
PIG-2855: Provide a method to measure time spent in UDFs (dvryaboy)
PIG-2837: AvroStorage throws StackOverFlowError (cheolsoo via sms)
PIG-2856: AvroStorage doesn't load files in the directories when a glob pattern matches both files and directories. (cheolsoo via sms)
PIG-2569: Fix org.apache.pig.test.TestInvoker.testSpeed (aklochkov via dvryaboy)
PIG-2858: Improve PlanHelper to allow finding any PhysicalOperator in a plan (dvryaboy)
PIG-2854: AvroStorage doesn't work with Avro 1.7.1 (cheolsoo via sms)
PIG-2779: Refactoring the code for setting number of reducers (jay23jack via billgraham)
PIG-2765: Implementing RollupDimensions UDF and adding ROLLUP clause in CUBE operator (prasanth_j via dvryaboy)
PIG-2814: Fix issues with Sample operator documentation (prasanth_j via dvryaboy)
PIG-2817: Documentation for Groovy UDFs (herberts via julien)
PIG-2492: AvroStorage should recognize globs and commas (cheolsoo via sms)
PIG-2706: Add clear to list of grunt commands (xalan via azaroth)
PIG-2823: TestPigContext.testImportList() does not pass if another javac in on the PATH (julien)
PIG-2800: pig.additional.jars path separator should align with File.pathSeparator instead of being hard-coded to ":" (jgordon via azaroth)
PIG-2797: Tests should not create their own file URIs through string concatenation, should use Util.generateURI instead (jgordon via azaroth)
PIG-2820: relToAbsolutePath is not replayed properly when Grunt reparses the script after PIG-2699 (julien)
PIG-2763: Groovy UDFs (herberts via julien)
PIG-2780: MapReduceLauncher should break early when one of the jobs throws an exception (jay23jack via daijy)
PIG-2804: Remove "PIG" exec type (dvryaboy)
PIG-2726: Handling legitimate NULL values in Cube operator (prasanth_j via dvryaboy)
PIG-2808: Add *.project to .gitignore (azaroth)
PIG-2787: change the module name in ivy to lowercase to match the maven repo (julien)
PIG-2632: Create a SchemaTuple which generates efficient Tuples via code gen (jcoveney)
PIG-2750: add artifacts to the ivy.xml for other jars Pig generates (julien)
PIG-2748: Change the names of the jar produced in the build folder to match maven conventions (julien)
PIG-2770: Allow easy inclusion of custom build targets (julien)
PIG-2697: pretty print schema via pig.pretty.print.schema (rangadi via jcoveney)
PIG-2673: Allow Merge join to follow an ORDER statement (dvryaboy)
PIG-2699: Reduce the number of instances of Load and Store Funcs down to 2+1 (julien)
PIG-2166: UDFs to join a bag (hluu via daijy)
PIG-2651: Provide a much easier to use accumulator interface (jcoveney via daijy)
PIG-2658: Add pig.script.submitted.timestamp and pig.job.submitted.timestamp in generated Map-Reduce job conf (billgraham)
PIG-2735: Add a pig.version.suffix property in build.xml to easily override with a build number (julien)
PIG-2705: outputSchema modification from scripting UDFs (levyjoshua via julien)
PIG-2724: Make Tuple Iterable (jcoveney)
PIG-2733: Add *.patch, *.log, *.orig, *.rej, *.class to gitignore (jcoveney)
PIG-2732: Let's get rid of the deprecated Tuple methods (jcoveney)
PIG-2638: Optimize BinInterSedes treatment of longs (jcoveney)
PIG-2727: PigStorage Source tagging does not need pig.splitCombination to be turned off (prkommireddi via dvryaboy)
PIG-2710: Implement Naive CUBE operator (prasanth_j via dvryaboy)
PIG-2714: Pig documentation on TOP funcation has issues (daijy)
PIG-2066: Accumulators should be able to early-terminate (jcoveney)
PIG-2600: Better Map support (prkommireddi via jcoveney)
PIG-2711: e2e harness: cache benchmark results between test runs (thw via daijy)
PIG-2702: Make Pig local mode (and tests) faster by working around the hard coded sleep(5000) in hadoop's JobControl (julien)
PIG-2659: add source location of the aliases in the physical plan (julien)
PIG-2547: Easier UDFs: Convenient EvalFunc super-classes (billgraham, dvryaboy)
PIG-2639: Utils.getSchemaFromString should automatically give name to all types, but fails on boolean (jcoveney)
PIG-2696: Enhance Job Stat to print out median map and reduce time (hluu via daijy)
PIG-2583: Add Grunt command to list the statements in cache (xalan via daijy)
PIG-2688: Log the aliases being processed for the current job (ddaniels888 via azaroth)
PIG-2680: TOBAG output schema reporting (andy schlaikjer via jcoveney)
PIG-2685: Fix error in EvalFunc ctor when implementing Algebraic UDF whose return type is parameterized (andy schlaikjer via jcoveney)
PIG-2664: Allow PPNL impls to get more job info during the run (billgraham)
PIG-2663: Expose helpful ScriptState methods (billgraham)
PIG-2660: PPNL notified of plan before it gets executed (billgraham)
PIG-2574: Make reducer estimator plugable (billgraham)
PIG-2677: Add target to build.xml to generate clover summary reports (gates)
PIG-2650: Convenience mock Loader and Storer to simplify unit testing of Pig scripts (julien)
PIG-2257: AvroStorage doesn't recognize schema_file field when JSON isn't used in the constructor (billgraham)
PIG-2587: Compute LogicalPlan signature and store in job conf (billgraham)
PIG-2619: HBaseStorage constructs a Scan with cacheBlocks = false
PIG-2604: Pig should print its build info at runtime (traviscrawford via dvryaboy)
PIG-2573: Automagically setting parallelism based on input file size does not work with HCatalog (traviscrawford via julien)
PIG-2538: Add helper wrapper classes for StoreFunc (billgraham via dvryaboy)
PIG-2010: registered jars on distributed cache (traviscrawford and julienledem via dvryaboy)
PIG-2533: Pig MR job exceptions masked on frontend (traviscrawford via dvryaboy)
PIG-2525: Support pluggable PigProcessNotifcationListeners on the command line (dvryaboy)
PIG-2515: [piggybank] Make CustomFormatToISO return null on Exception in parsing dates (rjurney via dvryaboy)
PIG-2503: Make @MonitoredUDF inherited (dvryaboy)
PIG-2488: Move Python unit tests to e2e tests (alangates via daijy)
PIG-2456: Pig should have a pigrc to specify default script cache (prkommireddi via daijy)
PIG-2496: Cache resolved classes in PigContext (dvryaboy)
PIG-2482: Integrate HCat DDL command into Pig (daijy)
PIG-2479: changingPattern should be used with checkmodified in ivysettings.xml (abayer via azaroth)
PIG-2349: Ant build repeats ivy-buildJar several times (azaroth)
PIG-2359: Support more efficient Tuples when schemas are known (dvryaboy)
PIG-2282: Automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy (azaroth via daijy)
PIG-2468: Speed up TestBuiltin (dvryaboy)
PIG-2467: Speed up TestCommit (dvryaboy)
PIG-2460: Use guava 11 instead of r06 (dvryaboy)
PIG-2267: Make the name of the columns in schema optional (jcoveney via daijy)
PIG-2453: Fetching schema can be very slow for multi-thousand file LOADs (dvryaboy)
PIG-2443: [Piggybank] Add UDFs to check if a String is an Integer And if a String is Numeric (prkommireddi via daijy)
PIG-2437: Use Ivy to get automaton.jar (azaroth)
PIG-2448: Convert more tests to use LOCAL mode (dvryaboy)
PIG-2438: Do not hardcode commons-lang version in build.xml (azaroth)
PIG-2422: Add log messages for Jython schema definitions (vivekp via gates)
PIG-2403: Reduce code duplication in SUM, MAX, MIN udfs (dvryaboy)
PIG-2245: Add end to end test for tokenize (markroddy via gates)
PIG-2327: bin/pig doesn't have any hooks for picking up ZK installation deployed from tarballs (rvs via hashutosh)
PIG-2382: Modify .gitignore to ignore pig-withouthadoop.jar (azaroth via hashutosh)
PIG-2380: Expose version information more cleanly (jcoveney via azaroth)
PIG-2311: STRSPLIT needs to allow bytearray arguments (xuting via olgan)
PIG-2365: Current TOP implementation needlessly results in a null bag name (jcoveney via dvryaboy)
PIG-2151: Add annotation to specify output schema in Java UDFs (dvryaboy)
PIG-2230: Improved error message for invalid parameter format (xuitingz via olgan)
PIG-2328: Add builtin UDFs for building and using bloom filters (gates)
PIG-2338: Need signature for EvalFunc (daijy)
PIG-2337: Provide UDF with input schema (xutingz via daijy)
OPTIMIZATIONS
BUG FIXES
PIG-3147: Spill failing with "java.lang.RuntimeException: InternalCachedBag.spill() should not be called" (knoguchi via dvryaboy)
PIG-3109: Missing license headers (jarcec via cheolsoo)
PIG-3022: TestRegisteredJarVisibility.testRegisteredJarVisibility fails with hadoop-2.0.x (rohini via cheolsoo)
PIG-3125: Fix zebra compilation error (cheolsoo)
PIG-3051: java.lang.IndexOutOfBoundsException failure with LimitOptimizer + ColumnPruning (knoguchi via rohini)
PIG-3076: make TestScalarAliases more reliable (julien)
PIG-3020: "Duplicate uid in schema" error when joining two relations derived from the same load statement (jcoveney)
PIG-3044: hotfix to remove divide by 0 error (jcoveney)
PIG-3033: test-patch failed with javadoc warnings (fang fang chen via cheolsoo)
PIG-3058: Upgrade junit to at least 4.8 (fang fang chen via cheolsoo)
PIG-2978: TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x (cheolsoo)
PIG-3039: Not possible to use custom version of jackson jars (rohini)
PIG-3045: Specifying sorting field(s) at nightly.conf - fix sortArgs (rohini via cheolsoo)
PIG-2979: Pig.jar doesn't work with hadoop-2.0.x (cheolsoo)
PIG-3035: With latest version of hadoop23 pig does not return the correct exception stack trace from backend (rohini)
PIG-2405: some unit test case failed with open JDK (fang fang chen via cheolsoo)
PIG-3018: Refactor TestScriptLanguage to remove duplication and write script in different files (julien)
PIG-2973: TestStreaming test times out (cheolsoo)
PIG-3001: TestExecutableManager.testAddJobConfToEnv fails randomly (cheolsoo)
PIG-3017: Pig's object serialization should use compression (jcoveney)
PIG-2968: ColumnMapKeyPrune fails to prune a subtree inside foreach (knoguchi via cheolsoo)
PIG-2999: Regression after PIG-2975: BinInterSedesTupleRawComparator secondary sort failing (knoguchi via azaroth)
PIG-2998: Fix TestScriptLanguage and TestMacroExpansion (cheolsoo via jcoveney)
PIG-2975: TestTypedMap.testOrderBy failing with incorrect result (knoguchi via jcoveney)
PIG-2950: Fix tiny documentation error in BagToString builtin (initialcontext via daijy)
PIG-2967: Fix Glob_local test failure for Pig E2E Test Framework (sushantj via daijy)
PIG-1283: COUNT on null bag causes failure (analog.sony via jcoveney)
PIG-2958: Pig tests do not appear to have a logger attached (daijyc via jcoveney)
PIG-2926: TestPoissonSampleLoader failing on rhel environment (jcoveney)
PIG-2985: TestRank1,2,3 fail with hadoop-2.0.x (rohini via azaroth)
PIG-2971: Add new parameter to specify the streaming environment (jcoveney)
PIG-2963: Illustrate command and POPackageLite (cheolsoo via jcoveney)
PIG-2961: BinInterSedesRawComparator broken by TUPLE_number patch (jcoveney)
PIG-2932: Setting high default_parallel causes IOException in local mode (cheolsoo via gates)
PIG-2737: [piggybank] TestIndexedStorage is failing, should be refactored (jcoveney)
PIG-2935: Catch NoSuchMethodError when StoreFuncInterface's new cleanupOnSuccess method isn't implemented. (gates via dvryaboy)
PIG-2920: e2e tests override PERL5LIB environment variable (azaroth)
PIG-2917: SpillableMemoryManager memory leak for WeakReference (haitao.yao via dvryaboy)
PIG-2938: All unit tests that use MR2 MiniCluster are broken in trunk (cheolsoo via dvryaboy)
PIG-2936: Tuple serialization bug (jcoveney)
PIG-2930: ant test doesn't compile in trunk (cheolsoo via daijy)
PIG-2791: Pig does not work with ViewFileSystem (rohini via daijy)
PIG-2833: org.apache.pig.pigunit.pig.PigServer does not initialize set default log level of pigContext (cheolsoo via jcoveney)
PIG-2744: Handle Pig command line with XML special characters (lulynn_2008 via daijy)
PIG-2637: Command-line option -e throws TokenMgrError exception (lulynn_2008 via daijy)
PIG-2887: Macro cannot handle negative number (knoguchi via gates)
PIG-2844: ant makepom is misconfigured (julien)
PIG-2896: Pig does not fail anymore if two macros are declared with the same name (julien)
PIG-2848: TestBuiltInBagToTupleOrString fails now that mock.Storage enforces not overwriting output (julien)
PIG-2884: JobControlCompiler mis-logs after reducer estimation (billgraham)
PIG-2876: Bump up Xerces version (jcoveney)
PIG-2866: PigServer fails with macros without a script file (billgraham)
PIG-2860: [piggybank] TestAvroStorageUtils.testGetConcretePathFromGlob fails on some version of hadoop (cheolsoo via jcoveney)
PIG-2861: PlanHelper imports org.python.google.common.collect.Lists instead of org.google.common.collect.Lists (jcoveney)
PIG-2849: Errors in document Getting Started (miyakawataku via billgraham)
PIG-2843: Typo in Documentation (eric59 via billgraham)
PIG-2841: Inconsistent URL in Docs (eric59 via billgraham)
PIG-2740: get rid of "java[77427:1a03] Unable to load realm info from SCDynamicStore" log lines when running pig tests (julien)
PIG-2839: mock.Storage overwrites output with the last relation written when storing UNION (julien)
PIG-2840: Fix SchemaTuple bugs (jcoveney)
PIG-2842: TestNewPlanOperatorPlan fails when new Configuration() picks up a previous minicluster conf file (julien)
PIG-2827: Unwrap exception swallowing in TOP (haitao.yao via jcoveney)
PIG-2825: StoreFunc signature setting in LogicalPlan broken (jcoveney)
PIG-2815: class loader management in PigContext (rangadi via jcoveney)
PIG-2813: Fix test regressions from PIG-2632 (jcoveney)
PIG-2806: Fix merge join test regression from PIG-2632 (jcoveney)
PIG-2809: TestUDFContext broken by PIG-2699 (julienledem via daijy)
PIG-2807: TestParser TestPigStorage TestNewPlanOperatorPlan broken by PIG-2699 (julienledem via daijy)
PIG-2782: Specifying sorting field(s) at nightly.conf (cheolsoo via daijy)
PIG-2790: After Pig-2699 the script schema (LOAD ... USING ... AS {script schema}) is passed after getSchema is called (daijy)
PIG-2777: Docs are broken due to malformed xml after PIG-2673 (dvryaboy)
PIG-2593: Filter by a boolean value does not work (jay23jack via daijy)
PIG-2665: Bundled Jython jar in Pig 0.10.0-RC breaks module import in Python scripts with embedded Pig Latin (daijy)
PIG-2736: Support implicit cast from bytearray to boolean (jay23jack via daijy)
PIG-2508: PIG can unpredictably ignore deprecated Hadoop config options (thw via dvryaboy)
PIG-2691: Duplicate TOKENIZE schema (jay23jack via azaroth)
PIG-2173: piggybank datetime conversion javadocs not properly formatted (hluu via daijy)
PIG-2709: PigAvroRecordReader should specify which file has a problem when throwing IOException (mpercy via daijy)
PIG-2640: Usage message gives wrong information for Pig additional jars (prkommireddi via daijy)
PIG-2652: Skew join and order by don't trigger reducer estimation (dvryaboy)
PIG-2616: JobControlCompiler.getInputSizeFromLoader must handle exceptions from LoadFunc.getStatistics (billgraham)
PIG-2644: Piggybank's HadoopJobHistoryLoader throws NPE when reading broken history file (herberts via daijy)
PIG-2627: Custom partitioner not set when POSplit is involved in Plan (aniket486 via daijy)
PIG-2596: Jython UDF does not handle boolean output (aniket486 via daijy)
PIG-2649: org.apache.pig.parser.ParserValidationException does not expose the cause exception
PIG-2540: [piggybank] AvroStorage can't read schema on amazon s3 in elastic mapreduce (rjurney via jcoveney)
PIG-2618: e2e local fails to build
PIG-2608: Typo in PigStorage documentation for source tagging (prkommireddi via daijy)
PIG-2590: running ant tar and rpm targets on same copy of pig source results in problems (thejas)
PIG-2581: HashFNV inconsistent/non-deterministic due to default platform encoding (prkommireddi via daijy)
PIG-2514: REGEX_EXTRACT not returning correct group with non greedy regex (romainr via daijy)
PIG-2532: Registered classes fail deserialization in frontend (traviscrawford via julien)
PIG-2549: org.apache.pig.piggybank.storage.avro - Broken documentation link for AvroStorage (chrisas via daijy)
PIG-2322: varargs functions do not get passed the arguments in Python embedding (julien)
PIG-2491: Pig docs still mention hadoop-site.xml (daijy)
PIG-2504: Incorrect sample provided for REGEX_EXTRACT (prkommireddi via daijy)
PIG-2502: Make "hcat.bin" configurable in e2e test (daijy)
PIG-2501: Changes needed to contrib/piggybank/java/build.xml in order to build piggybank.jar with Hadoop 0.23
(ekoontz via daijy)
PIG-2499: Pig TestGrunt.testShellCommand occasionally fails (tomwhite via daijy)
PIG-2326: Pig minicluster tests can not be run from eclipse (julienledem via daijy)
PIG-2432: Eclipse .classpath file is out of date (gates)
PIG-2427: getSchemaFromString throws away the name of the tuple that is in a bag (jcoveney via dvryaboy)
PIG-2425: Aggregate Warning does not work as expected on Embedding Pig in Java 0.9.1 (prkommireddi via thejas)
PIG-2384: Generic Invokers should use PigContext to resolve classes (dvryaboy)
PIG-2379: Bug in Schema.getPigSchema(ResourceSchema rSchema) improperly adds two level access (jcoveney via dvryaboy)
PIG-2355: ant clean does not clean e2e test build artifacts (daijy)
PIG-2352: e2e test harness' use of environment variables causes unintended effects between tests (gates)
Release 0.10.1 - Unreleased
BUG FIXES
PIG-3107: bin and autocomplete are missing in src release (daijy)
PIG-3106: Missing license header in several java file (daijy)
PIG-3099: Pig unit test fixes for TestGrunt(1), TestStore(2), TestEmptyInputDir(3) (vikram.dixit via daijy)
PIG-2953: "which" utility does not exist on Windows (daijy)
PIG-2960: Increase the timeout for unit test (daijy)
PIG-2942: DevTests, TestLoad has a false failure on Windows (jgordon via daijy)
PIG-2801: grunt "sh" command should invoke the shell implicitly instead of calling exec directly with the command tokens
(jgordon via daijy)
PIG-2798: pig streaming tests assume interpreters are auto-resolved (jgordon via daijy)
PIG-2795: Fix test cases that generate pig scripts with "load " + pathStr to encode "\" in the path (jgordon via daijy)
PIG-2796: Local temporary paths are not always valid HDFS path names (jgordon via daijy)
Release 0.10.0
INCOMPATIBLE CHANGES
IMPROVEMENTS
PIG-2664: Allow PPNL impls to get more job info during the run (billgraham)
PIG-2663: Expose helpful ScriptState methods (billgraham)
PIG-2660: PPNL notified of plan before it gets executed (billgraham)
PIG-2574: Make reducer estimator plugable (billgraham)
PIG-2601: Additional document for 0.10 (daijy)
PIG-2317: Ruby/Jruby UDFs (jcoveney via daijy)
PIG-1270: Push limit into loader (daijy)
PIG-2589: Additional e2e test for 0.10 new features (daijy)
PIG-2182: Add more append support to DataByteArray (gsingers via daijy)
PIG-438: Handle realiasing of existing Alias (A=B;) (daijy)
PIG-2548: Support for providing parameters to python script (daijy)
PIG-2518: Add ability to clean ivy cache in build.xml (daijy)
PIG-2300: Pig Docs - release 0.10.0 (and 0.9.1) (chandec via daijy)
PIG-2332: JsonLoader/JsonStorage (daijy)
PIG-2334: Set default number of reducers for S3N filesystem (ddaniels888 via daijy)
PIG-1387: Syntactical Sugar for PIG-1385 (azaroth)
PIG-2305: Pig should log the split locations in task logs (vivekp via thejas)
PIG-2293: Pig should support a more efficient merge join against data sources that natively support point
lookups or where the join is against large, sparse tables (aklish via daijy)
PIG-2287: add test cases for limit and sample that use expressions with
constants only (no scalar variables) (thejas via gates)
PIG-2092: Missing sh command from Grant shell (olgan)
PIG-2163: Improve nested cross to stream one relation (zjshen via daijy)
PIG-2249: Enable pig e2e testing on EC2 (gates)
PIG-2256: Upgrade Avro dependency to 1.5.3 (tucu00 via dvryaboy)
PIG-604: Kill the Pig job should kill all associated Hadoop Jobs (daijy)
PIG-2096: End to end tests for new Macro feature (gates)
PIG-2242: Allow the delimiter to be specified when calling TOKENIZE (markroddy via hashutosh)
PIG-2240: Allow any compression codec to be specified in AvroStorage (tomwhite via dvryaboy)
PIG-2229: Pig end-to-end tests should test local mode as well as mr mode (gates)
PIG-2235: Several files in e2e tests aren't being run (gates)
PIG-2196: Test harness should be independent of Pig (hashutosh) -- Missed few
changes in last commit.
PIG-2196: Test harness should be independent of Pig (hashutosh)
PIG-1429: Add Boolean Data Type to Pig (zjshen via daijy)
PIG-2218: Pig end-to-end tests should be accessible from top level build.xml (gates)
PIG-2176: add logical plan assumption checker (thejas)
PIG-1631: Support to 2 level nested foreach (aniket486 via daijy)
PIG-2191: Reduce amount of log spam generated by UDFs (dvryaboy)
PIG-2200: Piggybank cannot be built from the Git mirror (dvryaboy)
PIG-2168: CubeDimensions UDF (dvryaboy)
PIG-2189: e2e test harness needs to use Pig as a source of truth (gates via daijy)
PIG-1904: Default split destination (azaroth via thejas)
PIG-2143: Make PigStorage optionally store schema; improve docs. (dvryaboy)
PIG-1973: UDFContext.getUDFContext usage of ThreadLocal pattern
is not typical (woody via thejas)
PIG-2053: PigInputFormat uses class.isAssignableFrom() where
instanceof is more appropriate (woody via thejas)
PIG-2161: TOTUPLE should use no-copy tuple creation (dvryaboy)
PIG-1946: HBaseStorage constructor syntax is error prone (billgraham via dvryaboy)
PIG-2001: DefaultTuple(List) constructor is inefficient, causes List.size()
System.arraycopy() calls (though they are 0 byte copies),
DefaultTuple(int) constructor is a bit misleading wrt time
complexity (woody via thejas)
PIG-1916: Nested cross (zjshen via daijy)
PIG-2121: e2e test harness should use ant instead of make (gates)
PIG-2142: Allow registering multiple jars from DFS via single statement (rangadi via dvryaboy)
PIG-1926: Sample/Limit should take scalar (azaroth via thejas)
PIG-1950: e2e test harness needs to be able to compare to previous version of
Pig (gates)
PIG-536: the shell script 'pig' does not work if PIG_HOME has the word 'hadoop' in it's directory (miguno via olgan)
PIG-2108 e2e test harness needs to be able to mark certain tests as ignored
(gates)
PIG-1825: ability to turn off the write ahead log for pig's HBaseStorage (billgraham via dvryaboy)
PIG-1772: Pig 090 Documentation (chandec via olgan)
PIG-1772: Pig 090 Documentation (chandec via olgan)
PIG-1772: Pig 090 Documentation (chandec via olgan)
PIG-1824: Support import modules in Jython UDF (woody via rding)
PIG-1994: e2e test harness deployment implementation for existing cluster
(gates)
PIG-2036: [piggybank] Set header delimiter in PigStorageSchema (mmoeller via dvryaboy)
PIG-1949: e2e test harness should use bin/pig rather than calling java
directly (gates)
PIG-2026: e2e tests in eclipse classpath (azaroth via hashutosh)
PIG-2024: Incorrect jar paths in .classpath template for eclipse (azaroth via hashutosh)
OPTIMIZATIONS
PIG-2011: Speed up TestTypedMap.java (dvryaboy)
PIG-2228: support partial aggregation in map task (thejas)
BUG FIXES
PIG-2940: HBaseStorage store fails in secure cluster (cheolsoo via daijy)
PIG-2821: HBaseStorage should work with secure hbase (rohini via daijy)
PIG-2859: Fix few e2e test failures (rohini via daijy)
PIG-2729: Macro expansion does not use pig.import.search.path - UnitTest borked (johannesch via daijy)
PIG-2783: Fix Iterator_1 e2e test for Hadoop 23 (rohini via daijy)
PIG-2761: With hadoop23 importing modules inside python script does not work (rohini via daijy)
PIG-2759: Typo in document "Built In Functions" (daijy)
PIG-2745: Pig e2e test RubyUDFs fails in MR mode when running from tarball (cheolsoo via daijy)
PIG-2741: Python script throws an NameError: name 'Configuration' is not defined in case cache dir is not created
(knoguchi via daijy)
PIG-2669: Pig release should include pig-default.properties after rebuild (daijy)
PIG-2739: PyList should map to Bag automatically in Jython (daijy)
PIG-2730: TFileStorage getStatistics incorrectly throws an exception instead of returning null (traviscrawford via daijy)
PIG-2717: Tuple field mangled during flattening (daijy)
PIG-2721: Wrong output generated while loading bags as input (knoguchi via daijy)
PIG-2578: Multiple Store-commands mess up mapred.output.dir. (daijy)
PIG-2623: Support S3 paths for registering UDFs (nshkrob via daijy)
PIG-2505: AvroStorage won't read any file not ending in .avro (russell.jurney via daijy)
PIG-2585: Enable ignored e2e test cases (daijy)
PIG-2563: IndexOutOfBoundsException: while projecting fields from a bag (daijy)
PIG-2411: AvroStorage UDF in PiggyBank fails to STORE a bag of single-field tuples as Avro arrays (russell.jurney via daijy)
PIG-2565: Support IMPORT for macros stored in S3 Buckets (daijy)
PIG-2570: LimitOptimizer fails with dynamic LIMIT argument (daijy)
PIG-2543: PigStats.isSuccessful returns false if embedded pig script has sh commands (daijy)
PIG-2509: Util.getSchemaFromString fails with java.lang.NullPointerException when a tuple in a bag has no name (as when used in MongoStorage UDF) (jcoveney via daijy)
PIG-2559: Embedded pig in python; invoking sys.exit(0) causes script failure (vivekp via daijy)
PIG-2530: Reusing alias name in nested foreach causes incorrect results (daijy)
PIG-2489: Input Path Globbing{} not working with PigStorageSchema or PigStorage('\t', '-schema') (daijy)
PIG-2484: Fix several e2e test failures/aborts for 23 (daijy)
PIG-2400: Document has based aggregation support (chandec via daijy)
PIG-2444: Remove the Zebra *.xml documentation files from the TRUNK and Branch-10 (chandec via daijy)
PIG-2430: An EvalFunc which overrides getArgToFuncMapping with FuncSpec
with constructor arguments is not properly instantiated with said arguments (jcoveney via thejas)
PIG-2457: JsonLoaderStorage tests is broken for e2e (daijy)
PIG-2426: ProgressableReporter.progress(String msg) is an empty function (vivekp via daijy)
PIG-2363: _logs for streaming commands bug in new parser (vivekp via daijy)
PIG-2331: BinStorage in LOAD statement failing when input has curly braces (xutingz via thejas)
PIG-2391: Bzip_2 test is broken (xutingz via daijy)
PIG-2358: JobStats.getHadoopCounters() is never set and always returns null (xutingz via daijy)
PIG-2184: Not able to provide positional reference to macro invocations (xutingz via daijy)
PIG-2209: JsonMetadata fails to find schema for glob paths (daijy)
PIG-2165: Need a way to deal with params and param_file in embedded pig in python (daijy)
PIG-2313: NPE in ILLUSTRATE trying to get StatusReporter in STORE (daijy)
PIG-2335: bin/pig does not work with bash 3.0 (azaroth)
PIG-2275: NullPointerException from ILLUSTRATE (daijy)
PIG-2119: DuplicateForEachColumnRewrite makes assumptions about the position of LOGGenerate in the plan (daijy)
PIG-2290: TOBAG wraps tuple parameters in another tuple (ryan.hoegg via thejas)
PIG-2288: Pig 0.9 error message not useful as compared to 0.8 in case
of group by (vivekp via thejas)
PIG-2309: Keyword 'NOT' is wrongly treated as a UDF in split statement (vivekp via thejas)