bug: Error 1030 occurred in the query.

[haitaoguan]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Please confirm if bug report does NOT exists already ?

• I confirm there is no existing issue for this

Describe the problem

mysql> show create table bill_items\G
ERROR 1030 (HY000): Unknown error 1030

mysql> select count(*) from bill_items;
ERROR 1030 (HY000): Unknown error 1030

# perror 1030
MySQL error code 1030 (ER_GET_ERRNO): Got error %d from storage engine

###tianmu.log
[2023-05-25 09:34:17.859232] [29722] [ERROR] [engine.cpp:2287] MSG: Failed to create table share: bad DPN index file: ./lxjxc_production/bill_items.tianmu/columns/33
[2023-05-25 09:34:17.863178] [29722] [ERROR] [column_share.cpp:185] MSG: sorted beg: -------------------
[2023-05-25 09:34:17.863192] [29722] [ERROR] [column_share.cpp:187] MSG:      128  [0, 16384]
[2023-05-25 09:34:17.863199] [29722] [ERROR] [column_share.cpp:187] MSG:      129  [0, 21]
[2023-05-25 09:34:17.863205] [29722] [ERROR] [column_share.cpp:189] MSG: sorted end: -------------------
[2023-05-25 09:34:17.872613] [29722] [WARN] [exception.cpp:42] MSG: Exception: bad DPN index file: ./lxjxc_production/bill_items.tianmu/columns/33.

STACK TRACE BEGIN
        /opt/stonedb57/install/bin/mysqld() [0x9945d3]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::TableShare::TableShare(std::experimental::filesystem::v1::path const&, TABLE_SHARE const*)+0xde4) [0x1609cf4]
        /opt/stonedb57/install/bin/mysqld(Tianmu::core::Engine::GetTableShare(TABLE_SHARE const*)+0x174) [0x15daac4]
        /opt/stonedb57/install/bin/mysqld(Tianmu::DBHandler::ha_tianmu::open(char const*, int, unsigned int)+0x57) [0x1651d67]
        /opt/stonedb57/install/bin/mysqld(handler::ha_open(TABLE*, char const*, int, int)+0x33) [0xa7fb43]
        /opt/stonedb57/install/bin/mysqld(open_table_from_share(THD*, TABLE_SHARE*, char const*, unsigned int, unsigned int, unsigned int, TABLE*, bool)+0x763) [0x105e383]
        /opt/stonedb57/install/bin/mysqld(open_table(THD*, TABLE_LIST*, Open_table_context*)+0x1431) [0xf77301]
        /opt/stonedb57/install/bin/mysqld(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*)+0x4f2) [0xf78a92]
        /opt/stonedb57/install/bin/mysqld(open_tables_for_query(THD*, TABLE_LIST*, unsigned int)+0x48) [0xf79458]
        /opt/stonedb57/install/bin/mysqld() [0xfc13bd]
        /opt/stonedb57/install/bin/mysqld(mysql_execute_command(THD*, bool)+0x4b73) [0xfc8b03]
        /opt/stonedb57/install/bin/mysqld(mysql_parse(THD*, Parser_state*)+0x425) [0xfcad25]
        /opt/stonedb57/install/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1599) [0xfcc329]
        /opt/stonedb57/install/bin/mysqld(do_command(THD*)+0x207) [0xfcd047]
        /opt/stonedb57/install/bin/mysqld(handle_connection+0x2b8) [0x1085a38]
        /opt/stonedb57/install/bin/mysqld(pfs_spawn_thread+0x164) [0x155cf74]
        /lib64/libpthread.so.0(+0x7ea5) [0x7f9bd2a53ea5]
        /lib64/libc.so.6(clone+0x6d) [0x7f9bcfd8db0d]
STACK TRACE END

Expected behavior

No response

How To Reproduce

No response

Environment

./mysqld Ver 5.7.36-StoneDB-v1.0.1 for Linux on x86_64 (build-)
build information as follow:
Repository address: https://github.com/stoneatom/stonedb.git:stonedb-5.7-dev
Branch name: stonedb-5.7-dev
Last commit ID: 69a49bd
Last commit time: Date: Fri May 19 17:38:27 2023 +0800
Build time: Date: Tue May 23 10:54:18 CST 2023

Are you interested in submitting a PR to solve the problem?

• Yes, I will!

[RingsC]

The MySQL 1030 error occurs due to the local disk space issue. The error typically occurs when the server MySQL runs from, utilises the total disk space and tries to execute any of the below mentioned tasks.

Pls, check the free disk space.

[RingsC]

In function, void ColumnShare::scan_dpn(common::TX_ID xid) does the DPN index file check.

  // make sure the data is good
  auto second = segs.cbegin();
  for (auto first = second++; second != segs.cend(); ++first, ++second) {
    if (second->offset < first->offset + first->len) {
      TIANMU_LOG(LogCtl_Level::ERROR, "sorted beg: -------------------");
      for (auto &it : segs) {
        TIANMU_LOG(LogCtl_Level::ERROR, "     %u  [%ld, %ld]", it.idx, it.offset, it.len);
      }
      TIANMU_LOG(LogCtl_Level::ERROR, "sorted end: -------------------");
      throw common::DatabaseException("bad DPN index file: " + m_path.string());
    }
  }

The data of DPN index is here:

[2023-05-25 09:34:17.863178] [29722] [ERROR] [column_share.cpp:185] MSG: sorted beg: -------------------
[2023-05-25 09:34:17.863192] [29722] [ERROR] [column_share.cpp:187] MSG:      128  [0, 16384]
[2023-05-25 09:34:17.863199] [29722] [ERROR] [column_share.cpp:187] MSG:      129  [0, 21]
[2023-05-25 09:34:17.863205] [29722] [ERROR] [column_share.cpp:189] MSG: sorted end: -------------------

Which means it has 2 segs,
1st : index: 128; offset: 0, len: 16384
2nd: index: 129; offset: 0, len: 21

The first dpn length is 16384, and the offset of the second one is 0, that is impossible.

   len1        len2
|________|_________|

offset1   offset2

The offset of second one should become from end of the first one. 

the structure of seg

  struct seg {
    uint64_t offset;
    uint64_t len;
    common::PACK_INDEX idx;
  };

f4fc 3f00 0000 0000 f7fc 3f00 0000 0000
4000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 ffff ffff ffff ff7f
0000 0000 0000 0080 0000 0000 0000 0000
0000 0000 0000 0000 1540 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
4000 0000 4100 0000 4200 0000 4300 0000
4400 0000 4500 0000 4600 0000 8100 0000
4800 0000 4900 0000 4a00 0000 4b00 0000
4c00 0000 4d00 0000 4e00 0000 4f00 0000
5000 0000 5100 0000 5200 0000 5300 0000
5400 0000 5500 0000 5600 0000 5700 0000
5800 0000 5900 0000 5a00 0000 5b00 0000
5c00 0000 5d00 0000 5e00 0000 5f00 0000
6000 0000 6100 0000 6200 0000 6300 0000
6400 0000 6500 0000 6600 0000 6700 0000
6800 0000 6900 0000 6a00 0000 6b00 0000
6c00 0000 6d00 0000 6e00 0000 6f00 0000
7000 0000 7100 0000 7200 0000 7300 0000
7400 0000 7500 0000 7600 0000 7700 0000
7800 0000 7900 0000 7a00 0000 7b00 0000
7c00 0000 7d00 0000 7e00 0000 8000 0000

Related reading materials:
Ref: 1: https://db.cs.cmu.edu/mmap-cidr2022/ . It may involve a lot of insertions, deletions and updates, etc. it will make it's hard to identify correctness and performance problems.

[RingsC]

In open_table phashe, tianmu, firstly, will initialize all the shares of columns of the table. And, ColumnShare::Init, in this function, it will read the meta data from corresponding directories, and scan all dpn files to check it validation.

void ColumnShare::Init(common::TX_ID xid) {
  map_dpn();

  read_meta();

  scan_dpn(xid);
}

the map_dpn uses mmap to mapping the dnp file into memory, and then read_meta reads the meta info from that address. after that do dpns validation check.

[RingsC]

From the error message, bad DPN index file: ./lxjxc_production/bill_items.tianmu/columns/33., the error occured at initiializing the 34th column share. Therefore, it would be the best way to have a copy of columns/33/v/xxxx, it will help us to identify the root cause.

[RingsC]

Txn IDs in colid\v are in the first 34 columns:
646dbecc000028f8, 646dbecc000028fa, 646dbecc000028fc, 646dbecc000028fe, 646dca1f0000604b.

but int col 35, the trx ids are :
646dbecc000028f8, 646dbecc000028fa, 646dbecc000028fc ,646dbecc000028fe, 646dca1f000061f3. Obviously, in col 35, the transaction id changed, from 646dca1f0000604b to 646dca1f000061f3. From one transaction to another one.

some key infos :

(gdb) p xid.ToString()
$2 = "646dca1f0000604b"

(gdb) p hdr.numOfPacks
$6 = 64

(gdb) p hdr
$5 = {
  numOfRecords = 4193524,
  numOfNulls = 4193527,
  numOfPacks = 64,
  numOfDeleted = 0,
  auto_inc = 0,
  min = 9223372036854775807,
  max = -9223372036854775808,
  dict_ver = 0,
  unique = 0,
  unique_updated = 0,
  natural_size = 0,
  compressed_size = 16405
}

(gdb) p segs
$4 = std::__cxx11::list = {
  [0] = {
    offset = 0,
    len = 16384,
    idx = 128
  },
  [1] = {
    offset = 0,
    len = 21,
    idx = 129
  }
}

(gdb) p this->capacity
$7 = 512

The capacity equals capacity = sb.st_size / sizeof(DPN);. sb.st_size is size of DN file.

[RingsC]

There are some places, which maybe will change the data of DPN.

• Insert TianmuAttr::LoadData
• Update TianmuAttr::UpdateData
• Delete TianmuAttr::DeleteData

[RingsC]

By removed the deleted dpn from metadata. we have this.

mysql> select count(*) from bill_items; 
+----------+
| count(*) |
+----------+
|  4193521 |
+----------+
1 row in set (0.02 sec)